Being an emerging technology, robotic manipulation has encountered tremendous advancements due to technological developments starting from using sensors to artificial intelligence. Over the decades, robotic manipulation has advanced in terms of the versatility and flexibility of mobile robot platforms. Thus, robots are now capable of interacting with the world around them. To interact with the real world, robots require various sensory inputs from their surroundings, and the use of vision is rapidly increasing nowadays, as vision is unquestionably a rich source of information for a robotic system. In recent years, robotic manipulators have made significant progress towards achieving human-like abilities. There is still a large gap between human and robot dexterity, especially when it comes to executing complex and long-lasting manipulations. This paper comprehensively investigates the state-of-the-art development of vision-based robotic application, which includes the current state, components, and approaches used along with the algorithms with respect to the control and application of robots. Furthermore, a comprehensive analysis of those vision-based applied algorithms, their effectiveness, and their complexity has been enlightened here. To conclude, there is a discussion over the constraints while performing the research and potential solutions to develop a robust and accurate vision-based robot manipulation.

Upper limb dysfunction (ULD) is common following a stroke, spinal cord injury, trauma, and occupational accidents. Post-stroke patients with ULD need long-term assistance from therapists for their rehabilitation, which generally occurs at the hospital or outpatient clinic. Physical therapists are unavailable because of geographical, financial, and scheduling concerns, and continuity of care needs to be improved due to the need to travel to multiple locations for therapy. As a result, providing specific, tailored therapy programs is challenging due to the absence of feedback and real-time monitoring. An effective telerehabilitation system can address this issue and is more cost-effective for healthcare providers and patients than traditional inpatient or person-to-person rehabilitation. Remotely operating robotic devices and using advanced technology improves patient and healthcare provider safety and reduces injuries. In this study, we developed a novel telerehabilitation framework for rehabilitation robots utilizing PTC’s Industrial Internet of Things (IIoT) platform to remotely provide robot-aided therapies for individuals with ULD. With the developed telerehabilitation framework, an operator can teleoperate the rehab robots to deliver Upper-limb (UL) exercises via an Augmented Reality (AR) based graphical user interface (GUI). This AR platform communicates bidirectionally using ThingWorx IIOT. It leverages the digital twin (DT) structure facilitated by Vuforia studio to visualize the physical robot motions happening in remote places. The telerehabilitation framework was validated through a commercially available robot (xArm 5), an exoskeleton (SREx), and an end-effector type rehabilitation robot (DMRbot) developed at Biorobotics Lab, UWM. The experiment results show that the telerehabilitation system can successfully provide UL rehab exercises in 2D and 3D planes via AR. The proposed framework is developed to facilitate robust and more promising robot-aided rehabilitation sessions remotely, and it can also be applied in other medical applications.

The increasing prevalence of upper and lower extremity (ULE) functional deficiencies presents a significant challenge, as it restricts individuals’ ability to perform daily tasks independently. Robotic devices are emerging as assistive devices to assist individuals with limited ULE functionalities in activities of daily living (ADLs). While assistive manipulators are available, manual control through traditional methods like joysticks can be cumbersome, particularly for individuals with severe hand impairments and vision limitations. Therefore, autonomous/semi-autonomous control of a robotic assistive device to perform any ADL task is open to research. This study addresses the necessity of fostering independence in ADLs by proposing a creative approach. We present a vision-based control system for a six-degrees-of-freedom (DoF) robotic manipulator designed for semi-autonomous “pick-and-place” tasks, one of the most common activities among ADLs. Our approach involves selecting and training a deep-learning-based object detection model with a dataset of 47 ADL objects, forming the base for a 3D ADL object localization algorithm. The proposed vision-based control system integrates this localization technique to identify and manipulate ADL objects (e.g., apples, oranges, capsicums, and cups) in real time, returning them to specific locations to complete the “pick-and-place” task. Experimental validation involving an xArm6 (six DoF) robot from UFACTORY in diverse settings demonstrates the system’s adaptability and effectiveness, achieving an overall 72.9% success rate in detecting, localizing, and executing ADL tasks. This research contributes to the growing field of autonomous assistive devices, enhancing independence for individuals with functional impairments.

The advancements in mixed reality (MR) technology in recent years have provided us with excellent prospects for creating novel approaches to supplement conventional physiotherapy to maintain a sufficient quantity and quality of rehabilitation. The use of MR systems to facilitate patients’ participation in intensive, repetitive, and task-oriented practice using cutting-edge technologies to enhance functionality and facilitate recovery is very encouraging. Multiple studies have found that patients who undergo therapy using MR experience significant improvements in upper limb function; however, assessing the efficacy of MR is challenging due to the wide variety of methods and tools used. Because of these challenges, a novel approach, gamified MR-based solution for upper extremity rehabilitation, is proposed, which is an MR application for the Microsoft HoloLens 2, complete with game levels, and can measure the ranges of motion of the arm joints. The proposed rehabilitative system’s functionality and usability were evaluated with ten healthy adult participants with no prior arm-related injuries and two occupational therapists (OTs). The system successfully provided rehab exercises for upper limb injuries through interactive mixed-reality games. The system can mimic upper limb behavior without additional sensors during rehab sessions. Unlike previously researched technologically-based rehabilitation methods, this method can integrate arm–joint data within the application and are independent of one another. The results and comparisons show that this system is relevant, accurate, and superior to previous VR-based rehabilitation methods because the VR-based system is blind to the surroundings, whereas the proposed approach has spatial awareness of the environment.

A significant number of individuals in the United States use assistive devices to enhance their mobility, and a considerable portion of those who depend on such aids require assistance from another individual in performing daily living activities. The introduction of robotic grippers has emerged as a transformative intervention, significantly contributing to the cultivation of independence. However, there are few grippers in the fields, which help with mimicking human hand-like movements (mostly grasping and pinching, with adoptive force control) to grasp and carry objects. Additionally, the data are not available even on how many Activities of Daily Living (ADL) objects they can handle. The goal of the research is to offer a new three-fingered gripper for daily living assistance, which can both grasp and pinch with adaptive force, enabling the capabilities of handling wide-ranging ADL objects with a minimal footprint. It is designed to handle 90 selective essential ADL objects of different shapes (cylindrical, irregular, rectangular, and round), sizes, weights, and textures (smooth, rough, bumpy, and rubbery). The gripper boasts a meticulously engineered yet simple design, facilitating seamless manufacturing through 3D printing technology without compromising its operational efficacy. The gripper extends its functionality beyond conventional grasping, featuring the capability to pinch (such as holding a credit card) and securely hold lightweight objects. Moreover, the gripper is adaptable to grasping various objects with different shapes and weights with controlled forces. In evaluation, the developed gripper went through rigorous load tests and usability tests. The results demonstrated that the users picked and placed 75 objects out of 90 daily objects. The gripper held and manipulated objects with dimensions from 25 mm to 80 mm and up to 2.9 kg. For heavy-weight objects (like books) where the centroid is far apart from the grasping areas, it is difficult to hold them due to high torque. However, objects’ textures have no significant effect on grasping performance. Users perceived the simplicity of the gripper. Further investigation is required to assess the utility and longevity of grippers. This study contributes to developing assistive robots designed to enhance object manipulation, thereby improving individuals’ independence and overall quality of life.

Background: Applications of robotics in daily life are becoming essential by creating new possibilities in different fields, especially in the collaborative environment. The potentials of collaborative robots are tremendous as they can work in the same workspace as humans. A framework employing a top-notch technology for collaborative robots will surely be worthwhile for further research. Objective: This study aims to present the development of a novel framework for the collaborative robot using mixed reality. Methods: The framework uses Unity and Unity Hub as a cross-platform gaming engine and project management tool to design the mixed reality interface and digital twin. It also uses the Windows Mixed Reality platform to show digital materials on holographic display and the Azure mixed reality services to capture and expose digital information. Eventually, it uses a holographic device (HoloLens 2) to execute the mixed reality–based collaborative system. Results: A thorough experiment was conducted to validate the novel framework for mixed reality–based control of a collaborative robot. This framework was successfully applied to implement a collaborative system using a 5–degree of freedom robot (xArm-5) in a mixed reality environment. The framework was stable and worked smoothly throughout the collaborative session. Due to the distributed nature of cloud applications, there is a negligible latency between giving a command and the execution of the physical collaborative robot. Conclusions: Opportunities for collaborative robots in telerehabilitation and teleoperation are vital as in any other field. The proposed framework was successfully applied in a collaborative session, and it can also be applied in other similar potential applications for robust and more promising performance.