Navigating robots with precision in complex environments remains a significant challenge. In this article, we present an innovative approach to enhance robot localization in dynamic and intricate spaces like homes and offices. We leverage Visual Question Answering (VQA) techniques to integrate semantic insights into traditional mapping methods, formulating a novel position hypothesis generation to assist localization methods, while also addressing challenges related to mapping accuracy and localization reliability. Our methodology combines a probabilistic approach with the latest advances in Monte Carlo Localization methods and Visual Language models. The integration of our hypothesis generation mechanism results in more robust robot localization compared to existing approaches. Experimental validation demonstrates the effectiveness of our approach, surpassing state-of-the-art multi-hypothesis algorithms in both position estimation and particle quality. This highlights the potential for accurate self-localization, even in symmetric environments with large corridor spaces. Furthermore, our approach exhibits a high recovery rate from deliberate position alterations, showcasing its robustness. By merging visual sensing, semantic mapping, and advanced localization techniques, we open new horizons for robot navigation. Our work bridges the gap between visual perception, semantic understanding, and traditional mapping, enabling robots to interact with their environment through questions and enrich their map with valuable insights. The code for this project is available on GitHub https://github.com/juandpenan/topology_nav_ros2.

Robots that execute language-conditioned tasks in dynamic environments often rely on feedback only after an action has failed, which can be insufficient when failures involve collisions or workspace conflicts. This paper presents a predictive monitoring framework that uses Vision-Language Models (VLMs) to assess near-future execution risk during robot task execution. The framework first generates structured plans with action execution conditions and a plan-level fallback action. During execution, a monitoring module combines visual observations, the current action, and the relevant execution conditions to estimate whether a condition is likely to be violated within a short future time window. When the predicted risk exceeds a task-specific threshold, the robot halts the current action, executes the fallback behavior, and replans from the updated state. We evaluate the approach in Gazebo simulation on mobile navigation with a moving human obstacle and manipulation with two robot arms sharing a workspace. Across controlled collision-risk settings, the proposed method achieves higher task success rates than reactive VLM-based baselines while requiring fewer replanning events than conservative current-state precondition checking. The results indicate that predictive vision-language monitoring can improve task completion in simulated dynamic robot tasks, while remaining subject to limitations such as VLM latency, prompt sensitivity, and evaluation beyond simulation.

This research introduces a novel approach to enhance cooperation among robots by collaboratively executing Behavior Trees (BTs). A new node, the DelegateActionNode, allows task delegation as subtrees, while a shared virtual memory (blackboard) enables efficient coordination. This method supports distributed behavior execution, even with potential connectivity loss. Experiments with real robots demonstrate effective virtual memory sharing, robust performance under stress, and 100% success in BT delegation. The approach shows significant potential for advancing multi-robot systems and their cooperative task capabilities.

This dataset was collected during the 2023 and 2024 RoboCup competitions using the TIAGo robot equipped with an RGB-D camera, a Hokuyo laser, and a RODE microphone. The dataset includes ROSbag files that capture the robot's sensory data and planning task behavior, as well as video recordings that provide third-person perspectives of task execution. These data provide information about the performance of autonomous robots in social tasks and navigation in dynamic environments with human interaction.

In recent years, planning frameworks have enabled the creation and execution of plans in robots using classical planning approaches based on the Planning Domain Definition Language (PDDL). The dynamic nature of the environments in which these robots operate requires that execution plans adapt to new conditions, either by repairing plans to improve efficiency or because they are no longer valid. Determining the appropriate moment to initiate such repairs is the focus of our research. This paper presents a novel approach to this problem by using Large Language Models (LLMs) to make informed plan repair decisions during robot operation. Our approach introduces an LLM-based semantic evaluation heuristic that goes beyond the traditional heuristic methods employed in symbolic planning frameworks, while addressing the common hallucinations associated with task planning when relying solely on generative artificial intelligence. Our approach uses the semantic evaluation capabilities of LLMs to track environmental features and forecast hazards. This allows the system to proactively identify dangerous situations and adapt plans more efficiently. We experimentally demonstrate the validity of our approach using real robots in environments where both the environmental conditions and the goals to be achieved change dynamically.