In this work, we propose the first study of a technical validation of an assistive robotic platform, which has been designed to assist people with neurodevelopmental disorders. The platform is called LOLA2 and it is equipped with an artificial intelligence-based application to reinforce the learning of daily life activities in people with neurodevelopmental problems. LOLA2 has been integrated with an ROS-based navigation system and a user interface for healthcare professionals and their patients to interact with it. Technically, we have been able to embed all these modules into an NVIDIA Jetson Xavier board, as well as an artificial intelligence agent for online action detection (OAD). This OAD approach provides a detailed report on the degree of performance of a set of daily life activities that are being learned or reinforced by users. All the human–robot interaction process to work with users with neurodevelopmental disorders has been designed by a multidisciplinary team. Among its main features are the ability to control the robot with a joystick, a graphical user interface application that shows video tutorials with the activities to reinforce or learn, and the ability to monitor the progress of the users as they complete tasks. The main objective of the assistive robotic platform LOLA2 is to provide a system that allows therapists to track how well the users understand and perform daily tasks. This paper focuses on the technical validation of the proposed platform and its application. To do so, we have carried out a set of tests with four users with neurodevelopmental problems and special physical conditions under the supervision of the corresponding therapeutic personnel. We present detailed results of all interventions with end users, analyzing the usability, effectiveness, and limitations of the proposed technology. During its initial technical validation with real users, LOLA2 was able to detect the actions of users with disabilities with high precision. It was able to distinguish four assigned daily actions with high accuracy, but some actions were more challenging due to the physical limitations of the users. Generally, the presence of the robot in the therapy sessions received excellent feedback from medical professionals as well as patients. Overall, this study demonstrates that our developed robot is capable of assisting and monitoring people with neurodevelopmental disorders in performing their daily living tasks.

Dense video captioning involves detecting and describing events within video sequences. Traditional methods operate in an offline setting, assuming the entire video is available for analysis. In contrast, in this work we introduce a groundbreaking paradigm: Live Video Captioning (LVC), where captions must be generated for video streams in an online manner. This shift brings unique challenges, including processing partial observations of the events and the need for a temporal anticipation of the actions. We formally define the novel problem of LVC and propose innovative evaluation metrics specifically designed for this online scenario, demonstrating their advantages over traditional metrics. To address the novel complexities of LVC, we present a new model that combines deformable transformers with temporal filtering, enabling effective captioning over video streams. Extensive experiments on the ActivityNet Captions dataset validate the proposed approach, showcasing its superior performance in the LVC setting compared to state-of-the-art offline methods. To foster further research, we provide the results of our model and an evaluation toolkit with the new metrics integrated at: https://github.com/gramuah/lvc .

In this paper we address the problem of visual semantic navigation (VSN), in which a robot needs to navigate through an environment to reach an object having only access to egocentric RGB perception sensors. This is a recently explored problem, where most of the approaches leverage last advances in deep learning models for visual perception, combined with reinforcement learning (RL) strategies. Nonetheless, after a review of the literature, it is complicated to perform direct comparisons between the different solutions. The main difficulties lie in the fact that the navigation environments in which the experimental metrics are reported are not accessible, and each approach uses different RL libraries. In this paper, we release a publicly available experimental setup for the VSN problem, with the aim of providing a clear benchmark. It has been constructed using pyRIL, an open source python library for RL, and two navigation environments: Miniwolrd-Maze from gym-miniworld, and one 3D scene from HM3D dataset using AI Habitat simulator. We finally propose a state-of-the-art VSN model, consisting in a Contrastive Language Image Pretraining (CLIP) visual encoder plus a set of two recurrent neural networks for producing the discrete navigation actions. This model is evaluated in the proposed experimental setup, with a careful analysis of the main VSN challenges, namely: the sparse rewards problem; and the exploitation-exploration trade-off. Code is available at: https://github.com/gramuah/vsn.