Scene graph is a structured representation of a scene that can clearly express the objects, attributes, and relationships between objects in the scene. As computer vision technology continues to develop, people are no longer satisfied with simply detecting and recognizing objects in images; instead, people look forward to a higher level of understanding and reasoning about visual scenes. For example, given an image, we want to not only detect and recognize objects in the image, but also understand the relationship between objects (visual relationship detection), and generate a text description (image captioning) based on the image content. Alternatively, we might want the machine to tell us what the little girl in the image is doing (Visual Question Answering (VQA)), or even remove the dog from the image and find similar images (image editing and retrieval), etc. These tasks require a higher level of understanding and reasoning for image vision tasks. The scene graph is just such a powerful tool for scene understanding. Therefore, scene graphs have attracted the attention of a large number of researchers, and related research is often cross-modal, complex, and rapidly developing. However, no relatively systematic survey of scene graphs exists at present. To this end, this survey conducts a comprehensive investigation of the current scene graph research. More specifically, we first summarize the general definition of the scene graph, then conducte a comprehensive and systematic discussion on the generation method of the scene graph (SGG) and the SGG with the aid of prior knowledge. We then investigate the main applications of scene graphs and summarize the most commonly used datasets. Finally, we provide some insights into the future development of scene graphs.
15 March 2023 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Object detection (OD) is a crucial computer vision task that has seen the development of many algorithms and models over the years. While the performance of current OD models has improved, they have also become more complex, making them impractical for industry applications due to their large parameter size. To tackle this problem, knowledge distillation (KD) technology was proposed in 2015 for image classification and subsequently extended to other visual tasks due to its ability to transfer knowledge learned by complex teacher models to lightweight student models. This paper presents a comprehensive survey of KD-based OD models developed in recent years, with the aim of providing researchers with an overview of recent progress in the field. We conduct an in-depth analysis of existing works, highlighting their advantages and limitations, and explore future research directions to inspire the design of models for related tasks. We summarize the basic principles of designing KD-based OD models, describe related KD-based OD tasks, including performance improvements for lightweight models, catastrophic forgetting in incremental OD, small object detection, and weakly/semi-supervised OD. We also analyze novel distillation techniques, i.e. different types of distillation loss, feature interaction between teacher and student models, etc. Additionally, we provide an overview of the extended applications of KD-based OD models on specific datasets, such as remote sensing images and 3D point cloud datasets. We compare and analyze the performance of different models on several common datasets and discuss promising directions for solving specific OD problems.
29 July 2024 in IEEE Transactions on Communications
With integrated sensing and communications (ISAC) and reconfigurable intelligent surface (RIS) emerging as critical enablers for future mobile communications, their combination has attracted increasing attention lately. To effectively utilize RIS for improving ISAC, we propose two novel designs targeting different scenarios. The first design strikes for a spectral-efficient ISAC by seeking to maximize the weighted sum rate (WSR) of communications and minimize sensing radiation pattern approximation error. The second design aims to achieve an energy-efficient ISAC by optimizing the power allocation between communications and sensing subject to communication quality of service (QoS) constraints. Different optimization problems are formulated for the two designs, with practical constraints of RIS considered, including unit modulus and discrete phase shift. Efficient solutions are developed for the non-convex optimization problems by adeptly employing techniques including weighted minimum mean squared error (WMMSE), fractional programming (FP), second-order cone programming (SOCP), semi-definite relaxing (SDR), and feasibility check. Simulation results demonstrate the non-trivial improvements of WSR, communication energy efficiency and sensing radiation patterns achieved by the proposed designs, also highlighting their superiority over the conventional methods.
Wi-Fi CSI-based gait recognition is a non-intrusive passive biometric identification technology that has garnered significant attention in the fields of security and smart furniture due to its user-friendly nature. However, in practical application scenarios, gait recognition systems face the challenge of reliably identifying subjects across different scenes or states. To overcome this challenge, this paper proposes DCS-Gait, a domain adaptation solution for cross-scene and cross-state gait recognition based on Wi-Fi CSI. DCS-Gait leverages a novel data distribution measurement called Cross-Attention Metric to align the class-level data distribution differences, enabling the model to learn invariant features across scenes and states. To address the issue of data annotation, we employ a pre-training method to obtain pseudo labels for the dataset. Additionally, a combined matching filtering technique is utilized to generate high-quality pseudo labels for unrecognized data, which can be further employed for supervised model training. We evaluated the effectiveness of DCS-Gait on a large test set consisting of 34 subjects, 2 scenes, and 3 different states, and the results demonstrate significant improvements over the state-of-the-art baselines in both cross-scene and cross-state gait recognition tasks. DCS-Gait provides a promising and reliable solution for accurate cross-scene and cross-state gait recognition in real-world settings.
Wi-Fi signal-based person identification has become a hot research topic due to the widespread deployment of Wi-Fi devices and the fact that these approaches are noncontact, passive, and privacy-preserving. While the existing related methods and systems have achieved good performance for person identification, they also encounter many significant challenges in practical applications. Due to the propagation properties of Wi-Fi signals, the signal at the receiver will change significantly when the user’s appearance changes. This makes single-appearance trained models unusable for cross-appearance recognition tasks. To address this challenge, we propose a deep learning-based framework for appearance-independent identification using Wi-Fi signals (WiAi-ID), the core of which lies in the fact that the domain discriminator and feature extractor are trained together in an adversarial manner, thus forcing the model to extract identity-inherent features independent of human appearance, and introduces a multiscale CNN adaptation module to capture time-span-based features. We collected Wi-Fi signal data of pedestrians with different appearances. The experimental results show that WiAi-ID can effectively eliminate the impact on identification due to pedestrian appearance variations and accordingly outperforms the current state-of-the-art video and wireless signal-based recognition methods.