Home service robots operating indoors, such as inside houses and offices, require the real-time and accurate identification and location of target objects to perform service tasks efficiently. However, images captured by visual sensors while in motion states usually contain varying degrees of blurriness, presenting a significant challenge for object detection. In particular, daily life scenes contain small objects like fruits and tableware, which are often occluded, further complicating object recognition and positioning. A dynamic and real-time object detection algorithm is proposed for home service robots. This is composed of an image deblurring algorithm and an object detection algorithm. To improve the clarity of motion-blurred images, the DA-Multi-DCGAN algorithm is proposed. It comprises an embedded dynamic adjustment mechanism and a multimodal multiscale fusion structure based on robot motion and surrounding environmental information, enabling the deblurring processing of images that are captured under different motion states. Compared with DeblurGAN, DA-Multi-DCGAN had a 5.07 improvement in Peak Signal-to-Noise Ratio (PSNR) and a 0.022 improvement in Structural Similarity (SSIM). An AT-LI-YOLO method is proposed for small and occluded object detection. Based on depthwise separable convolution, this method highlights key areas and integrates salient features by embedding the attention module in the AT-Resblock to improve the sensitivity and detection precision of small objects and partially occluded objects. It also employs a lightweight network unit Lightblock to reduce the network’s parameters and computational complexity, which improves its computational efficiency. Compared with YOLOv3, the mean average precision (mAP) of AT-LI-YOLO increased by 3.19%, and the detection precision of small objects, such as apples and oranges and partially occluded objects, increased by 19.12% and 29.52%, respectively. Moreover, the model inference efficiency had a 7 ms reduction in processing time. Based on the typical home activities of older people and children, the dataset Grasp-17 was established for the training and testing of the proposed method. Using the TensorRT neural network inference engine of the developed service robot prototype, the proposed dynamic and real-time object detection algorithm required 29 ms, which meets the real-time requirement of smooth vision.

Up to this day, precise manipulation still relies on human hands, which benefits from the rich neural sensing in the fingertips. For robotic hands, force perception, recognizing the softness of objects, and detecting slip can significantly improve the grasping and manipulating abilities. However, it is quite challenging to embed sensors into the compact fingertip. Here, we propose a soft sensor to perceive external forces through the output change of a pressure transducer. The soft sensor also acts as fingertip with its physical airbag. The pressure transducer inside sensor detects the contacting force. In the experiments, two fingers grasped an object at three different speeds. The feedback data were input to a random forest algorithm to perceive the softness of the contacting surface and identify the object. The accuracy was experimentally proved to be greater than 90% and have good robustness. A prototype of a robot hand employing the proposed soft fingertips was developed, which experimentally demonstrated the grasping and resistant ability to external perturbations. The robot hand manipulated a square object to track a preset trajectory using two fingers and a classical PID controller, which significantly depends on the embedded soft sensor with its force feedback, flexibility, adaptability.

Soft fingertips have distinct intrinsic features that allow robotic hands to offer adjustable and manageable stiffness for grasping. The stability of the grasp is determined by the contact stiffness between the soft fingertip and the object. Within this work, we proposed a line vector representation method based on the Winkler Model and investigated the contact stiffness between soft fingertips and objects to achieve control over the gripping force and fingertip displacement of the gripper without the need for sensors integrated in the fingertip. First, we derived the stiffness matrix of the soft fingertip, analyzed the contact stiffness, and constructed the global stiffness matrix; then, we established the grasp stiffness matrix based on the contact stiffness model, allowing for the analysis and evaluation of the soft fingertip’s manipulating process. Finally, our experiment demonstrated that the variation in object orientation caused by external forces can indicate the contact force status between the fingertip and the object. This contact force status is determined by the contact stiffness. The position error between the theoretical work and tested data was less than 9%, and the angle error was less than 5.58%. The comparison between the theoretical contact stiffness and the experimental results at the interface indicate that the present model for the contact stiffness is appropriate and the theoretical contact stiffness is consistent with the experiment data.

The routine use of prosthetic hands significantly enhances amputees’ daily lives, yet it often introduces cognitive load and reduces reaction speed. To address this issue, we introduce a wearable semi-autonomous hierarchical control framework tailored for amputees. Drawing inspiration from the visual processing stream in humans, a fully autonomous bionic controller is integrated into the prosthetic hand control system to offload cognitive burden, complemented by a Human-in-the-Loop (HIL) control method. In the ventral-stream phase, the controller integrates multi-modal information from the user’s hand–eye coordination and biological instincts to analyze the user’s movement intention and manipulate primitive switches in the variable domain of view. Transitioning to the dorsal-stream phase, precise force control is attained through the HIL control strategy, combining feedback from the prosthetic hand’s sensors and the user’s electromyographic (EMG) signals. The effectiveness of the proposed interface is demonstrated by the experimental results. Our approach presents a more effective method of interaction between a robotic control system and the human.