It is widely believed the brain-inspired spiking neural networks have the capability of processing temporal information owing to their dynamic attributes. However, how to understand what kind of mechanisms contributing to the learning ability and exploit the rich dynamic properties of spiking neural networks to satisfactorily solve complex temporal computing tasks in practice still remains to be explored. In this article, we identify the importance of capturing the multi-timescale components, based on which a multi-compartment spiking neural model with temporal dendritic heterogeneity, is proposed. The model enables multi-timescale dynamics by automatically learning heterogeneous timing factors on different dendritic branches. Two breakthroughs are made through extensive experiments: the working mechanism of the proposed model is revealed via an elaborated temporal spiking XOR problem to analyze the temporal feature integration at different levels; comprehensive performance benefits of the model over ordinary spiking neural networks are achieved on several temporal computing benchmarks for speech recognition, visual recognition, electroencephalogram signal recognition, and robot place recognition, which shows the best-reported accuracy and model compactness, promising robustness and generalization, and high execution efficiency on neuromorphic hardware. This work moves neuromorphic computing a significant step toward real-world applications by appropriately exploiting biological observations.

Processing spatiotemporal data sources with both high spatial dimension and rich temporal information is a ubiquitous need in machine intelligence. Recurrent neural networks in the machine learning domain and bio-inspired spiking neural networks in the neuromorphic computing domain are two promising candidate models for dealing with spatiotemporal data via extrinsic dynamics and intrinsic dynamics, respectively. Nevertheless, these networks have disparate modeling paradigms, which leads to different performance results, making it hard for them to cover diverse data sources and performance requirements in practice. Constructing a unified modeling framework that can effectively and adaptively process variable spatiotemporal data in different situations remains quite challenging. In this work, we propose hybrid spatiotemporal neural networks created by combining the recurrent neural networks and spiking neural networks under a unified surrogate gradient learning framework and a Hessian-aware neuron selection method. By flexibly tuning the ratio between two types of neurons, the hybrid model demonstrates better adaptive ability in balancing different performance metrics, including accuracy, robustness, and efficiency on several typical benchmarks, and generally outperforms conventional single-paradigm recurrent neural networks and spiking neural networks. Furthermore, we evidence the great potential of the proposed network with a robotic task in varying environments. With our proof of concept, the proposed hybrid model provides a generic modeling route to process spatiotemporal data sources in the open world.

Visual perception systems increasingly rely on hybrid sensing modalities, including RGB and event cameras, to operate robustly under complex and extreme environmental conditions. However, what fundamentally determines the respective advantages of RGB frames and dynamic events remains unclear. To this end, we introduce MAD-Drone (Modality Advantage Dataset for Drone Perception), a multi-condition visual dataset built in a physics-based simulation framework with a high-fidelity event camera model to provide explicit modality-aligned definitions. This enables to map how modality advantages vary across data conditions and introduce several interpretable modality-aware metrics to explain these variations within a unified space. Given systematic understanding, we finally design a lightweight plug-in module alongside simple modality-specific preprocessing strategies to translate data-driven insights into modality usage. Over multiple real-world datasets, including DSEC, PKU-DAVIS-SOD, NeRDD and our physically-collected MAD-Drone, spanning different object categories, consistent performance improvements are demonstrated. Our findings reveal previously underappreciated patterns: modality advantage is not determined by coarse-grained condition labels or any monotonic superiority of event-based sensing, but emerges from coupled reliability conditions involving the temporal variation, illumination statistics, and foreground-background information density. Ultimately, this work provides a data-centric foundation for understanding, explaining, and exploiting fundamental modality advantages in frame and event hybrid visual data.