This article identifies and summarizes software tools and frameworks proposed in the Human–Robot Interaction (HRI) literature for developing extended reality (XR) experiences using game engines. This review includes primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can control or interact with real robotic platforms using devices that extend the user’s reality. The objective of this article is not to present an extensive list of applications and tools. Instead, we present recent, relevant, common, and accessible frameworks and software tools implemented in research articles published in high-impact robotics conferences and journals. For this, we searched papers published during a seven-years period between 2015 and 2022 in relevant databases for robotics (Science Direct, IEEE Xplore, ACM digital library, Springer Link, and Web of Science). Additionally, we present and classify the application context of the reviewed articles in four groups: social robotics, programming of industrial robots, teleoperation of industrial robots, and Human–Robot collaboration (HRC).

This paper presents our development of an intuitive teleoperation system for Toyota's Human Support Robot. While this robot has been widely used as an autonomous agent in the field of service robotics, we explore a possibility of this robot for the use of remote assistance of daily activities in home environments. For rapid prototyping an affordable teleoperation system, we adopt a commercially available VR device as a user interface and develop a Robot Operating System (ROS)-based control framework building on the software application programming interfaces (APIs). In particular, we implement a singularity robust inverse kinematics algorithm to achieve a coordinated whole-body motion of the robot for safe and intuitive operation. We empirically evaluate the practical feasibility of the proposed teleoperation system through various test tasks. We discuss the potential and limitations of the current development for its future improvement.

In this paper, toward robotic gait assistance, we investigate the feasibility of a cane-type assistive mobile robot accompanying the user autonomously through a clinical pilot experiment. As widely known, gait ability is important for all people to keep their quality of life. However, for people having weakened lower limbs such as elderly people, their postural sway during walking could be insuppressible and cause falling. To support the gait motion of elderly people, our group has been developing a series of hand-holding cane robots named Intelligent Cane. Such assistive robots are expected to remove barriers to the independent lives of elderly people. Recently, we have focused on the potential of a companion robot that follows the user ahead and can be touched or grasped for bracing whenever the user needs it. In order to demonstrate proof of our concept through an experiment with a motion capture system, we propose a user companion strategy that enables our cane robot to keep a constant relative distance between the robot and the user walking on a treadmill. We evaluate the accuracy of the user companion in an experiment where a user walks on a treadmill. Then, we conduct a clinical experiment with three healthy subjects walking on with the treadmill with our cane robot as a pilot study. Through the clinical experiment, we evaluate a postural stabilization effect of physical interaction with the robot and discuss the feasibility of our robotic gait assistance methodology.

In this letter, we introduce our approach to walking assistance for elderly adults through predictive optimization of gait assistive force. We focus on providing supportive interaction force to the user during walking with a robotic assistive device with an admittance controlled mobile base. Appropriate physical human-robot interaction (pHRI) could be beneficial in reducing the risk associated with immobility such as disuse syndrome by encouraging physical activities with proper assistance. We propose an optimization algorithm based on a model predictive control approach in order to provide desirable assistive forces according to the estimated user's state during walking. Using a simplified human gait model with a linear inverted pendulum, we formulate the optimization of the assistive forces as a linear quadratic programming problem that can be suitable for real-time pHRI. Numerical simulations and experimental results demonstrate the feasibility of our gait support strategy in achieving appropriate compliant interactions during walking, fall prevention, and suitable positioning for user companion.

An intelligent assistive robotic walker can be a tremendous help to patients with mobility disorders, e.g., hemiplegia patients. However, at the same time, a full-size assistive walker could be difficult to steer due to the reduced balancing ability of the patient. An intuitive interface can provide a promising solution to the low operability of such robotized assistive devices. Commonly developed gait assistance methodologies perform real-time adjustment of admittance model parameters in a forward direction. However, such a variable admittance model often accidentally affects the rotation behavior as well, which can lead to reducing the operability of the robotic device. To resolve such operational issues, we propose an in-Hand Admittance Controller (i-HAC) as a novel physical human-robot interaction interface for a robotic assistive walker. i-HAC is composed of multiple admittance models based on grasping feedback from sensitive robotic skin placed on both handlebars of a robotic assistive walker. This controller is formulated as a multiple-point-mass-damper system and can generate intuitive movement in both single and dual grasping situations. We conduct two experiments to demonstrate the benefits of i-HAC on a full-sized robotic walker in going forward with single grasping and rotating with dual grasping in comparison to the conventional admittance controller. Our experimental studies show that i-HAC potentially reduces the operation effort by 85% in the single grasping case and improves rotation consistency by 93% in the dual grasping case.