People’s interactions with each other form the social relations in society. Understanding human social relations in the public space is of great importance for supporting the public administrations. Recognizing social relations through visual data captured by remote sensing cameras is one of the most efficient ways to observe human interactions in a public space. Generally speaking, persons in the same scene tend to know each other, and the relations between person pairs are strongly correlated. The scene information in which people interact is also one of the important cues for social relation recognition (SRR). The existing works have not explored the correlations between the scene information and people’s interactions. The scene information has only been extracted on a simple level and high level semantic features to support social relation understanding are lacking. To address this issue, we propose a social relation structure-aware local–global model for SRR to exploit the high-level semantic global information of the scene where the social relation structure is explored. In our proposed model, the graph neural networks (GNNs) are employed to reason through the interactions (local information) between social relations and the global contextual information contained in the constructed scene-relation graph. Experiments demonstrate that our proposed local–global information-reasoned social relation recognition model (SRR-LGR) can reason through the local–global information. Further, the results of the final model show that our method outperforms the state-of-the-art methods. In addition, we have further discussed whether the global information contributes equally to different social relations in the same scene, by exploiting an attention mechanism in our proposed model. Further applications of SRR for human-observation are also exploited.

Identifying relationships between people from images is essential for studying social activities and interactions, and this has significant potential to further the understanding of human social behaviors. Existing image-based research mainly explores social relationships at the dyadic level, i.e., recognizing pairwise relationships based on visual features of persons, objects, and scenes and their logical constraints. Notably, social relational structures are hierarchically nested, i.e., individuals and dyads are nested within group structures, as indicated in the social relations model (SRM) of social psychology. However, existing computer vision-based studies fail to consider hierarchical nested structures, thus overlooking some of the most important interactions, which leads to poor relation reasoning. To improve the performance of reasoning neural networks, we propose a novel SRM framework for progressive graph reasoning (PGR) to explore social interactions. Specifically, we construct individual–dyad and dyad–group graphs to progressively explore the impact of individuals and groups on recognition of dyadic relationships. A transformer is utilized to fuse visual features and graph reasoning knowledge into a comprehensive representation of social relationships. We demonstrate the effectiveness of the proposed model based on PGR using several public datasets and perform extensive ablation studies to explore the reasons behind its superior performance. Experimental results demonstrate that our proposed model successfully predicts social relationships with higher accuracy than state-of-the-art methods. Codes and datasets are available at: https://github.com/tw-repository/PGRSRR .

Social relationships refer to the connections that exist between people and indicate how people interact in society. The effective recognition of social relationships is conducive to further understanding human behavioral patterns and thus can be vital for more complex social intelligent systems, such as interactive robots and health self-management systems. The existing works about social relation recognition (SRR) focus on extracting features on different scales but lack a comprehensive mechanism to orchestrate various features which show different degrees of importance. In this paper, we propose a new SRR framework, namely Multi-level Transformer-Based Social Relation Recognition (MT-SRR), for better orchestrating features on different scales. Specifically, a vision transformer (ViT) is firstly employed as a feature extraction module for its advantage in exploiting global features. An intra-relation transformer (Intra-TRM) is then introduced to dynamically fuse the extracted features to generate more rational social relation representations. Next, an inter-relation transformer (Inter-TRM) is adopted to further enhance the social relation representations by attentionally utilizing the logical constraints among relationships. In addition, a new margin related to inter-class similarity and a sample number are added to alleviate the challenges of a data imbalance. Extensive experiments demonstrate that MT-SRR can better fuse features on different scales as well as ameliorate the bad effect caused by a data imbalance. The results on the benchmark datasets show that our proposed model outperforms the state-of-the-art methods with significant improvement.

This article investigates prescribed-time neurodynamic optimization-based incremental model predictive control with visual servoing for standoff tracking of quadrotor UAVs. A vision-based Lyapunov guidance vector is employed to plan the trajectories of image feature points. An incremental model predictive visual servoing controller is proposed to track these trajectories. A zeroing neurodynamic optimization approach is developed to provide a robust solution for the controller, and its prescribed-time stability is further proven. The superiority and practicality of the proposed visual servoing standoff tracking approach for both static and moving targets are demonstrated through simulations and physical experiments.