We compare event-cameras with fast (global shutter) frame-cameras experimentally, asking: “What is the application domain, in which an event-camera surpasses a fast frame-camera?” Surprisingly, finding the answer has been difficult. Our methodology was to test event- and frame-cameras on generic computer vision tasks where event-camera advantages should manifest. We used two methods: (1) a controlled, cheap, and easily reproducible experiment (observing a marker on a rotating disk at varying speeds); (2) selecting one challenging practical ballistic experiment (observing a flying bullet having a ground truth provided by an ultra-high-speed expensive frame-camera). The experimental results include sampling/detection rates and position estimation errors as functions of illuminance and motion speed; and the minimum pixel latency of two commercial state-of-the-art event-cameras (ATIS, DVS240). Event-cameras respond more slowly to positive than to negative large and sudden contrast changes. They outperformed a frame-camera in bandwidth efficiency in all our experiments. Both camera types provide comparable position estimation accuracy. The better event-camera was limited by pixel latency when tracking small objects, resulting in motion blur effects. Sensor bandwidth limited the event-camera in object recognition. However, future generations of event-cameras might alleviate bandwidth limitations.

Multi-modal robot programming with natural language and demonstration is a promising technique for efficient teaching of manipulation tasks in industrial environments. In particular, with modern dual-arm robots designed to quickly take over tasks at typical industrial workbenches, the direct teaching of task sequences hardly utilizes the robots' capabilities. We therefore propose a two-staged approach that combines natural language instructions and demonstration with simultaneous task allocation and motion scheduling based on constraint programming. Instead of providing a task description and demonstrations that are replayed to a large extent, the user describes tasks to be scheduled with all relevant constraints and demonstrates relevant locations relative to workpieces and other objects. With explicitly stated constraints on the partial ordering of tasks, the solver allocates the tasks to the robot arms and schedules them in time while avoiding self-collisions and reducing the makespan in our experiment by 33%. The linguistic concepts of naming and grouping enable systematic reuse of sub-task ensembles. The proposed approach is evaluated with four variants of a gluing use-case from furniture assembly in user studies with ten participants. In these user studies, we observed a speed-up for the task definition of more than six times compared to a textual specification of the planning problems using the Python-based planner API.

This paper introduces a dataset for training and evaluating methods for 6D pose estimation of hand-held tools in task demonstrations captured by a standard RGB camera. Despite the significant progress of 6D pose estimation methods, their performance is usually limited for heavily occluded objects, which is a common case in imitation learning, where the object is typically partially occluded by the manipulating hand. Currently, there is a lack of datasets that would enable the development of robust 6D pose estimation methods for these conditions. To overcome this problem, we collect a new dataset (Imitrob) aimed at 6D pose estimation in imitation learning and other applications where a human holds a tool and performs a task. The dataset contains image sequences of nine different tools and twelve manipulation tasks with two camera viewpoints, four human subjects, and left/right hand. Each image is accompanied by an accurate ground truth measurement of the 6D object pose obtained by the HTC Vive motion tracking device. The use of the dataset is demonstrated by training and evaluating a recent 6D object pose estimation method (DOPE) in various setups. The dataset and code are publicly available at http://imitrob.ciirc.cvut.cz/imitrobdataset.php .

Having the ability of generating people images in arbitrary, yet admissible, pose is a crucial prerequisite for Autonomous Driving applications. Firstly, because the existing datasets are quite limited in the human pose variation and appearance. Secondly, because the strict safety requirements call for the ability of validation on rare situations. Generating realistically looking people images is very challenging problem due to various transformations of individual body parts [2,6] self occlusions etc. We propose a novel approach for person image generation. Our approach allows generating people images in a required pose, indicated by specific pose keypoints and deals with occlusions. We build on top of the recent prevailing success of Generative Adversarial Networks [10]. Our contributions comprise of the networks architecture, as well as the novel loss terms specifically designed to generate visually appealing pedestrians fitting the surrounding environment well.

Manipulating cluttered cables, hoses or ropes is challenging for both robots and humans. Humans often simplify these perceptually challenging tasks by pulling or pushing tangled cables and observing the resulting motions. We propose to use a similar interactive perception principle to aid robotic cable manipulation. A fundamental building block of such an endeavor is a cable motion segmentation method that densely labels moving cable image pixels. This paper presents MovingCables, a moving cable dataset, which we hope will motivate the development and evaluation of cable motion segmentation algorithms. The dataset consists of real-world image sequences automatically annotated with ground truth segmentation masks and optical flow. In addition, we propose a cable motion segmentation method and evaluate its performance on the new dataset. Code, dataset, and visualizations are available at https://github.com/holesond/movingcables and https://doi.org/10.5281/zenodo.11475246 .