Sign language recognition has been utilized in human–machine interactions, improving the lives of people with speech impairments or who rely on nonverbal instructions. Thanks to its higher temporal resolution, less visual redundancy information and lower energy consumption, the use of an event camera with a new dynamic vision sensor (DVS) shows promise with regard to sign language recognition with robot perception and intelligent control. Although previous work has focused on event camera-based, simple gesture datasets, such as DVS128Gesture, event camera gesture datasets inspired by sign language are critical, which poses a great impediment to the development of event camera-based sign language recognition. An effective method to extract spatio-temporal features from event data is significantly desired. Firstly, the event-based sign language gesture datasets are proposed and the data have two sources: traditional sign language videos to event stream (DVS_Sign_v2e) and DAVIS346 (DVS_Sign). In the present dataset, data are divided into five classification, verbs, quantifiers, position, things and people, adapting to actual scenarios where robots provide instruction or assistance. Sign language classification is demonstrated in spike neuron networks with a spatio-temporal back-propagation training method, leading to the best recognition accuracy of 77%. This work paves the way for the combination of event camera-based sign language gesture recognition and robotic perception for the future intelligent systems.

Inspired by biological vision mechanism, event‐based cameras have been developed to capture continuous object motion and detect brightness changes independently and asynchronously, which overcome the limitations of traditional frame‐based cameras. Complementarily, spiking neural networks (SNNs) offer asynchronous computations and exploit the inherent sparseness of spatio‐temporal events. Notably, event‐based pixel‐wise optical flow estimations calculate the positions and relationships of objects in adjacent frames; however, as event camera outputs are sparse and uneven, dense scene information is difficult to generate and the local receptive fields of the neural network also lead to poor moving objects tracking. To address these issues, an improved event‐based self‐attention optical flow estimation network (SA‐FlowNet) that independently uses criss‐cross and temporal self‐attention mechanisms, directly capturing long‐range dependencies and efficiently extracting the temporal and spatial features from the event streams is proposed. In the former mechanism, a cross‐domain attention scheme dynamically fusing the temporal‐spatial features is introduced. The proposed network adopts a spiking‐analogue neural network architecture using an end‐to‐end learning method and gains significant computational energy benefits especially for SNNs. The state‐of‐the‐art results of the error rate for optical flow prediction on the Multi‐Vehicle Stereo Event Camera (MVSEC) dataset compared with the current SNN‐based approaches is demonstrated.

Thanks to their event-driven asynchronous computing capabilities and low power consumption advantages, spiking neural networks (SNNs) show significant potential for computer vision tasks, especially in object detection. However, effective training methods and optimization mechanisms for SNNs remain underexplored. This study proposes two high accuracy SNNs for object detection, AMS_YOLO and AMSpiking_VGG, integrating neuronal variants and attention mechanisms. To enhance these proposed networks, we explore the impact of incorporating different neuronal variants.The results show that the optimization in the SNN’s structure with neuronal variants outperforms that in the attention mechanism for object detection. Compared to the state-of-the-art in the current SNNs, AMS_YOLO improved by 6.7% in accuracy on the static dataset COCO2017, and AMS_Spiking has improved by 11.4% on the dynamic dataset GEN1.

The event camera, a new bio-inspired vision sensor with low latency and high temporal resolution, has brought great potential and demonstrated a promising application in machine vision and artificial intelligence. Corner detection is a key step of object motion estimation and tracking. However, most existing event-based corner detectors, such as G-eHarris and Arc*, lead to a huge number of redundant or wrong corners, and cannot strike a balance between the accuracy and real-time performance, especially in complex scenes with high texture that require higher computational costs. To address these issues, we propose an asynchronous corner detection method: a double threshold filter with Sigmoid eHarris (DTFS-eHarris) and an asynchronous corner tracker. The main contributions are that a double threshold filter is designed to reduce the redundant events and the improved Sigmoid function is utilized to represent the Surface of Active Events ($Sigmoi{d}^{*}$-SAE). We selected four scenes—shapes, dynamic, poster and boxes—from the public event camera dataset DAVIS240C to compare with the existing state-of-the-art hybrid method; our method has shown more than a 10% reduction in false positive rate and a 5% and 20% improvement in accuracy and throughput, respectively. The evaluations indicate that DTFS-eHarris shows a significant improvement, especially in complex scenes. Thus, it is anticipated to enhance the real-time performance and feature detection accuracy for future robotic applications.

The MARSS is the first integration of soldier and machine optimized for maintenance. It is a lightweight, open architecture, wearable personal computer with object oriented software that controls and integrates plug-in measurement instrumentation, diagnostics processes, interactive electronic technical manuals, and logistic databases for the soldier. Input and output to the system is by means of a head-mounted microphone and a flat-panel active matrix display. MARSS has wireless Local Area Network (LAN) interaction with the weapon system data bus and members of the maintenance team, allowing transmission of fault or diagnostic data. It contains multimedia repair and replace instructions on small, high-density, removable PCMCIA disks. Flexible board technology provides soldier comfort. The open architecture allows easy upgrading of the hardware and software, including new tools as they become available. New battery technology consists of very small, flat, flexible batteries producing six hours operation.