It is estimated that at least 15 million people worldwide live with severe deaf-blindness, with many more experiencing varying degrees of deaf-blindness. The existing options of assistance are mostly limited to walking canes, guide dogs and human care. We propose a wearable device which harnesses a multi-antenna mmWave radar transceiver and a haptic feedback array for real time detection of a person moving within a scene. We present our findings from a series of workshops with participants classed with multi-sensory impairments (MSI), to demonstrate the relative success of this approach and its potential for integration into existing assistance for the MSI of the future.
In the field of detection and ranging, multiple complementary sensing modalities may be used to enrich information obtained from a dynamic scene. One application of this sensor fusion is in public security and surveillance, where efficacy and privacy protection measures must be continually evaluated. We present a novel deployment of sensor fusion for the discrete detection of concealed metal objects on persons whilst preserving their privacy. This is achieved by coupling off-the-shelf mmWave radar and depth camera technology with a novel neural network architecture that processes radar signals using convolutional Long Short-Term Memory (LSTM) blocks and depth signals using convolutional operations. The combined latent features are then magnified using deep feature magnification to reveal cross-modality dependencies in the data. We further propose a decoder, based on the feature extraction and embedding block, to learn an efficient upsampling of the latent space to locate the concealed object in the spatial domain through radar feature guidance. We demonstrate the ability to detect the presence and infer the 3D location of concealed metal objects. We achieve accuracies of up to 95% using a technique that is robust to multiple persons. This work provides a demonstration of the potential for cost-effective and portable sensor fusion with strong opportunities for further development.
Optical modalities for noninvasive imaging of the human brain hold promise to fill the technology gap between cheap and portable devices such as electroencephalography (EEG) and expensive high-resolution instruments such as functional magnetic resonance imaging (fMRI), with devices that have high sample rates, high spatial resolution, and are relatively inexpensive. A major bottleneck for the widespread adoption of optical brain reading devices in clinics and neuroscience studies is the low number of photons emerging from deep layers of the brain that restrict the sensitivity of these methods to a maximum of 4 cm below the scalp,1–3 corresponding to the outermost layer of the cerebral cortex (gyri). However, imaging paradigms have been demonstrated using highly scattered transmitted photons in thick diffusive materials,4,5 and it has been theoretically shown that photons experiencing similar scattering length scales as the diameter of the human head can carry imaging information.6 Therefore, extending current optical methods to extract information about deep brain regions that are currently inaccessible, e.g., cortical folds (sulci), midbrain, and deep regions of the cerebellum, may be feasible with careful consideration of sources and detectors and incorporating time-resolved photon counting information so long as photons are detected through the extreme number of attenuation lengths.
The guiding and transport of energy, for example, of electromagnetic waves, underpins many modern technologies, ranging from long-distance optical fibre telecommunications to on-chip optical processors. Traditionally, a mechanism is required that exponentially localizes the waves or particles in the confinement region, such as total internal reflection at a boundary. Here we introduce a waveguiding mechanism that relies on a different origin for the exponential confinement and that arises owing to the physics of diffusion. We demonstrate this concept using light and show that the photon density can propagate as a guided mode along a core structure embedded in a scattering opaque material, enhancing light transmission by orders of magnitude and along non-trivial, such as curved, trajectories. This waveguiding mechanism can also occur naturally, for example, in the cerebrospinal fluid surrounding the brain and along tendons in the human body, and is to be expected in other systems that follow the same physics such as neutron diffusion.
Accurate depth estimation is crucial for many fields, including robotics, navigation, and medical imaging. However, conventional depth sensors often produce low-resolution (LR) depth maps, making detailed scene perception challenging. To address this, enhancing LR depth maps to high-resolution (HR) ones has become essential, guided by HR-structured inputs like RGB or grayscale images. We propose a novel sensor fusion methodology for guided depth super-resolution (GDSR), a technique that combines LR depth maps with HR images to estimate detailed HR depth maps. Our key contribution is the Incremental guided attention fusion (IGAF) module, which effectively learns to fuse features from RGB images and LR depth maps, producing accurate HR depth maps. Using IGAF, we build a robust super-resolution model and evaluate it on multiple benchmark datasets. Our model achieves state-of-the-art results compared to all baseline models on the NYU v2 dataset for , , and upsampling. It also outperforms all baselines in a zero-shot setting on the Middlebury, Lu, and RGB-D-D datasets. Code, environments, and models are available on GitHub.
In the field of detection and ranging, multiple complementary sensing modalities may be used to enrich the information obtained from a dynamic scene. One application of this sensor fusion is in public security and surveillance, whose efficacy and privacy protection measures must be continually evaluated. We present a novel deployment of sensor fusion for the discrete detection of concealed metal objects on persons whilst preserving their privacy. This is achieved by coupling off-the-shelf mmWave radar and depth camera technology with a novel neural network architecture that processes the radar signals using convolutional Long Short-term Memory (LSTM) blocks and the depth signal, using convolutional operations. The combined latent features are then magnified using a deep feature magnification to learn cross-modality dependencies in the data. We further propose a decoder, based on the feature extraction and embedding block, to learn an efficient upsampling of the latent space to learn the location of the concealed object in the spatial domain through radar feature guidance. We demonstrate the detection of presence and inference of 3D location of concealed metal objects with an accuracy of up to 95%, using a technique that is robust to multiple persons. This work provides a demonstration of the potential for cost effective and portable sensor fusion, with strong opportunities for further development.