Pushing is one of the fundamental nonprehensile manipulation skills to impart to an object changes in position and orientation. To exploit this skill to manipulate novel objects, explicit knowledge of their physical properties should be given a priori. In this work, we estimate the center of mass (CoM) of an object by narrowing down its probable location with a deep learning model and Mason’s voting theorem. In addition, we propose the Zero Moment Two Edge Pushing (ZMTEP) method to translate a novel object without rotation to a goal pose. The proposed method enables a pusher to select the most suitable two-edge-contact configuration for a given object using the estimated CoM and the geometrical shape of the object. Notably, neither the friction between the object and its support plane nor the friction between the object and the pusher are assumed to be known. We evaluate the proposed CoM estimation and ZMTEP methods through a series of experiments in both simulation and real robotic pusher settings. The result shows that the CoM estimation method has good mean squared error properties and small standard deviation, and the ZMTEP method significantly outperforms competitive baseline methods. Note to Practitioners —This article aims to endow robotic arms with the capability of moving or aligning objects by pushing, which is much more simple and secure than pick-and-place or in-hand manipulations. Most in-demand manipulation skills require sophisticated hand design and control, which might not be affordable for industrial applications staying cost-competitive. In contrast, robot pushing can be implemented with different types of simple pushers and straightforwardly applied to pre-grasp manipulation. This article makes the estimation of an object’s CoM location practical. Building upon the estimation method, a robust and noise-tolerant two-edge-contact pushing configuration selection method is presented to translate an arbitrarily shaped unknown object to its goal pose.

Room acoustical parameters (RAPs), room geometrical parameters (RGPs) and instantaneous occupancy levels are essential metrics for parameterizing the room acoustical characteristics (RACs) of a sound field around a listener's local environment, offering comprehensive indications for various applications. Current blind estimation methods either fail to cover a broad range of real-world acoustic environments in the context of real background noise or estimate only a few RAPs and RGPs from noisy single-channel speech signals. In addition, they are limited in their ability to estimate the instantaneous occupancy level. In this paper, we propose BERP, a new universal approach to blindly estimate RAPs, RGPs, and occupancy levels. It consists of two modules: one for RAPs and RGPs and another for occupancy levels. For the former task, we use a shared room feature encoder that combines attention mechanisms with convolutional layers to learn common features across room parameters, and multiple separate parametric predictors for continuous estimation of each parameter in parallel. The combination of attention and convolutions enables the model to capture acoustic features both locally and globally from speech, yielding more robust and multitask generalizable common features. Separate predictors allow the model to independently optimize for each room parameter to reduce task learning conflict and improve per-task performance. This architecture enables universal and efficient estimation of room parameters while maintaining satisfactory performance. For occupancy level estimation, we reuse the identical encoder with a classification head, exploiting the encoder's strong sequencefeature extraction. To evaluate the effectiveness of the proposed approach, we compile a task-specific dataset from several publicly available datasets, including synthetic and real reverberant recordings. The results reveal that BERP achieves state-of-the-art (SOTA) performance and excellent adaptability to real-world scenarios. The code, datasets, and weights are available on GitHub.

Object rearrangement is widely demanded in many of the manipulation tasks performed by industrial and service robots. Rearranging an object through planar pushing is deemed energy efficient and safer compared with the pick-and-place operation. However, due to the unknown physical properties of the object, re-arranging an object toward the target position is difficult to accomplish. Even though robots can benefit from multi-modal sensory data for estimating novel object dynamics, the exact estimation error bound is still unknown. In this work, firstly, we demonstrate a way to obtain an error bound on the center of mass (CoM) estimation for the novel object only using a position-controlled robot arm and a vision sensor. Specifically, we extend Mason's Voting Theorem (MVT) to object CoM estimation in the absence of accurate information on friction and object shape. The probable CoM locations are monotonously narrowed down to a convex region, and the Extended Voting Theorems (EVT's) guarantee that the convex region contains the CoM ground truth in the presence of contact normal estimation error and pushing execution error. For the object translation task, existing methods generally assume that the pusher-object system's physical properties and full-state feedback are available, or utilize iterative pushing executions, which limits the application of planar pushing to real-world settings. In this work, assuming a nominal friction coefficient between the pusher and object through contact normal error bound analysis, we leverage the estimated convex region and the Zero Moment Two Edge Pushing (ZMTEP) method [1] to select the contact configurations for object pure translation. It is ensured that the selected contact configurations are capable of tolerating the CoM estimation error. The experimental results show that the object can be accurately translated to the target position with only two controlled pushes at most.

Integrating the Artificial Intelligence (AI) vision module into the robot grasping system can significantly improve its generalizability, thereby enhancing the efficiency of Human-Robot Interaction (HRI). However, the inherent lack of interpretability in AI also opens the gate to external threats. In this work, we reveal a novel safety risk in this vision-guided robot grasping system by proposing the Shortcut-enhanced Multimodal Backdoor Attack (SEMBA), which can manipulate the grasp quality score using the backdoor trigger leading to a misguided grasping sequence. The SEMBA may thus cause potentially hazardous grasping and pose a threat to human safety in HRI. Specifically, we initially present the Multimodal Shortcut Searching Algorithm (MSSA) to find the pixel value that deviates the most from the mean and standard deviation of the multimodal dataset, along with the pivotal pixel position for individual images. This will guarantee that the proposed attack is effective in complex, multi-class object scenarios. Next, based on MSSA, we devise the Multimodal Trigger Generator (MTG) to create diverse multimodal backdoor triggers and integrate them into the dataset, ensuring that our attack has the multimodality attribute. We conduct extensive experiments on the benchmark datasets and a cobot, showing the effectiveness of the proposed method both in the digital and physical worlds. Our demo videos are available in supplementary items. Note to Practitioners—Robot grasping systems are typically designed to be safe and reliable in HRI scenarios. However, integrating an AI-powered vision module into such systems can introduce substantial unpredictability, particularly when relying on third-party data. This work introduces a novel backdoor attack method to reveal a new safety risk in this vision-guided robot grasping system. Our method aims to mislead the robot into performing hazardous grasps by altering the grasping sequence. It emphasizes the attack’s effectiveness in complex, multi-class object scenarios and highlights its multimodal nature. To the best of our knowledge, this is the first study to present backdoor attacks in vision-guided robot grasping. Our approach paves the way toward future AI-powered visual grasping safety studies and provides valuable insights into building a more reliable and trustworthy vision-guided system for cobots.

Optical flow estimation is a fundamental and long-standing task in computer vision, facilitating the understanding of motion within visual scenes. In this study, we aim to improve optical flow estimation, particularly in challenging scenarios involving small and fast-moving objects. Specifically, we proposed a learning-based model incorporating two key components: the Hierarchical Motion Field Alignment module, which ensures accurate estimation of objects of varying sizes while maintaining manageable computational complexity, and the Correlation Self-Attention module, which effectively handles large displacements, making the model suitable for scenarios with fast-moving objects. Additionally, we introduced a Multi-Scale Correlation Search layer to enhance the four-dimensional cost volume, enabling the model to address various types of motion. Experimental results demonstrate that our model achieves superior generalization performance and significantly improves the estimation of small, fast-moving objects.