Occlusion and pose variations, which can change facial appearance significantly, are two major obstacles for automatic Facial Expression Recognition (FER). Though automatic FER has made substantial progresses in the past few decades, occlusion-robust and pose-invariant issues of FER have received relatively less attention, especially in real-world scenarios. This paper addresses the real-world pose and occlusion robust FER problem in the following aspects. First, to stimulate the research of FER under real-world occlusions and variant poses, we annotate several in-the-wild FER datasets with pose and occlusion attributes for the community. Second, we propose a novel Region Attention Network (RAN), to adaptively capture the importance of facial regions for occlusion and pose variant FER. The RAN aggregates and embeds varied number of region features produced by a backbone convolutional neural network into a compact fixed-length representation. Last, inspired by the fact that facial expressions are mainly defined by facial action units, we propose a region biased loss to encourage high attention weights for the most important regions. We validate our RAN and region biased loss on both our built test datasets and four popular datasets: FERPlus, AffectNet, RAF-DB, and SFEW. Extensive experiments show that our RAN and region biased loss largely improve the performance of FER with occlusion and variant pose. Our method also achieves state-of-the-art results on FERPlus, AffectNet, RAF-DB, and SFEW. Code and the collected test data will be publicly available.

Intelligent occupancy sensing is becoming a vital underpinning for various emerging applications in smart homes, such as security surveillance and human behavior analysis. However, prevailing approaches mainly rely on video camera, ambient sensors or wearable devices, which either requires arduous deployment or arouses privacy concerns. In this paper, we present a novel real-time, device-free and privacy-preserving WiFi-enabled IoT platform for occupancy sensing, which can promote a myriad of emerging applications. It is designed to achieve an optimal tradeoff between performance and scalability. Our system empowers commercial off-the-shelf (COTS) WiFi routers to collect Channel State Information (CSI) measurements and provides an efficient cloud server for computing via a lightweight communication protocol. To demonstrate the usefulness of our platform, an occupancy detection system is developed by exploiting the CSI curve of human presence. Furthermore, we also design an innovative activity recognition system based on our platform and machine learning techniques with high availability and extensibility. In the evaluation, the experimental results show that our platform enables these applications efficiently, with the accuracy of 96.8% and 90.6% in terms of occupancy detection and recognition respectively.

WiFi technology has been applied to various places due to the increasing requirement of high-speed Internet access. Recently, besides network services, WiFi sensing is appealing in smart homes since it is device free, cost effective and privacy preserving. Though numerous WiFi sensing methods have been developed, most of them only consider single smart home scenario. Without the connection of powerful cloud server and massive users, large-scale WiFi sensing is still difficult. In this article, we first analyze and summarize these obstacles, and propose an efficient large-scale WiFi sensing framework, namely, EfficientFi. The EfficientFi works with edge computing at WiFi access points and cloud computing at center servers. It consists of a novel deep neural network that can compress fine-grained WiFi channel state information (CSI) at edge, restore CSI at cloud, and perform sensing tasks simultaneously. A quantized autoencoder and a joint classifier are designed to achieve these goals in an end-to-end fashion. To the best of our knowledge, the EfficientFi is the first Internet of Things-cloud-enabled WiFi sensing framework that significantly reduces communication overhead while realizing sensing tasks accurately. We utilized human activity recognition (HAR) and identification via WiFi sensing as two case studies, and conduct extensive experiments to evaluate the EfficientFi. The results show that it compresses CSI data from 1.368 Mb/s to 0.768 kb/s with extremely low error of data reconstruction and achieves over 98% accuracy for HAR.

Indoor localization has attracted more and more attention because of its importance in many applications. One of the most popular techniques for indoor localization is the received signal strength indicator (RSSI) based fingerprinting approach. Since RSSI values are very complicated and noisy, conventional machine learning algorithms often suffer from limited performance. Recently developed deep learning algorithms have been shown to be powerful for the analysis of complex data. In this paper, we propose a local feature-based deep long short-term memory (LF-DLSTM) approach for WiFi fingerprinting indoor localization. The local feature extractor attempts to reduce the noise effect and extract robust local features. The DLSTM network is able to encode temporal dependencies and learn high-level representations for the extracted sequential local features. Real experiments have been conducted in two different environments, i.e., a research lab and an office. We also compare the proposed approach with some state-of-the-art methods for indoor localization. The results show that the proposed approach achieves the best localization performance with mean localization errors of 1.48 and 1.75 m under the research lab and office environments, respectively. The improvements of our proposed approach over the state-of-the-art methods range from $\text{18.98}\text{\%}$ to $\text{53.46}\text{\%}$ .

We propose a gesture recognition system that leverages existing WiFi infrastructures and learns gestures from Channel State Information (CSI) measurements. Having developed an innovative OpenWrt-based platform for commercial WiFi devices to extract CSI data, we propose a novel deep Siamese representation learning architecture for one-shot gesture recognition. Technically, our model extends the capacity of spatio-temporal patterns learning for the standard Siamese structure by incorporating convolutional and bidirectional recurrent neural networks. More importantly, the representation learning is ameliorated by our Siamese framework and transferable pairwise loss which helps to remove structured noise such as individual heterogeneity and various measurement conditions during domain-different training. Meanwhile, our Siamese model also enables one-shot learning for higher availability in reality. We prototype our system on commercial WiFi routers. The experiments demonstrate that our model outperforms state-of-the-art solutions for temporal-spatial representation learning and achieves satisfactory results under one-shot conditions.

Channel state information (CSI)-based human activity recognition (HAR) has received great attention in recent years due to its advantages in privacy protection, insensitivity to illumination, and no requirement for wearable devices. In this article, we propose a multimodal channel state information-based activity recognition (MCBAR) system that leverages existing WiFi infrastructures and monitors human activities from CSI measurements. MCBAR aims to address the performances degradation of WiFi-based human recognition systems due to environmental dynamics. Specifically, we address the issue of nonuniformly distributed unlabeled data with rarely performed activities by taking advantages of the generative adversarial network (GAN) and semisupervised learning. We apply a multimodal generator to approximate the CSI data distribution in different environment settings with limited measured CSI data. The generated CSI data using the multimodal generator can provide better diversity for knowledge transfer. This multimodal generator improves the ability of MCBAR to recognize specific activities with various CSI patterns caused by environmental dynamics. Compared to state-of-the-art CSI-based recognition systems, MCBAR is more robust as it is able to handle the nonuniformly distributed CSI data collected from a new environment setting. In addition, diverse generated data from the multimodal generator improves the stability of the system. We have tested MCBAR under multiple experimental settings at different places. The experimental results demonstrate that our algorithm overcomes environmental dynamics and outperforms existing HAR systems.