Accurate object detection and 6D pose estimation are the key technologies in robotic grasping applications, where efficiency and robustness are the two most desirable goals. Especially, for textureless industrial parts, it is difficult for most existing methods to extract robust image features from cluttered scenarios with heavy occlusion. To address this challenge, we propose a novel pixel-wise prediction strategy using local features to infer global information based on the inherent local–global relations of rigid objects. This strategy is robust to missing or disturbed local information since each pixel has an independent prediction, and the dense prediction manner can mitigate the instability caused by outliers. Accordingly, we first generate dense pixel-wise predictions of the object category, center, and keypoint from image features extracted by an encoder–decoder network. Then, these predictions are used to vote on and identify the keypoint locations of the specific instance object, and finally, the poses are estimated from the keypoints by an uncertainty perspective-n-point ( PnP ) algorithm. Experiments on various scenarios are implemented to illustrate the advantages of our approach on severe industrial scenes, and a robotic grasping platform is constructed to evaluate its application performance.
28 July 2023 in IEEE Transactions on Industrial Informatics
Estimating the 6-D pose of an object is a vital and challenging task for robot vision systems in industrial robotic grasping. With the wide use of 3-D cameras, the additional acquired depth image provides geometric information of the scene to increase the pose estimation performance but leads to a challenge, fully leveraging the two-modal data, the color image and the depth image. Previous works usually adopt two individual strategies to handle the data, which suffer from limited accuracy and efficiency since the two complementary data are not fully explored. Thus, we propose a depth adaptive feature extraction and dense prediction network that decouples the scale-dependent and the scale-invariant information from the depth image. The former guides the network to perceive the 3-D structure of the scene, and the latter, together with color image, provides the scene textures for feature extraction. The proposed network not only fuses multimodal textures but also retains their 3-D structure. In addition, a dense prediction strategy is adopted to regress the object pose; this approach can mitigate the instability caused by outliers. We conduct various evaluations on a real-world industrial dataset to illustrate the advantages of the proposed approach; and a practical robotic grasping platform is presented to demonstrate its application performance.
Urban design is a subject that is concerned with the shape, the surface and its physical arrangement of all kinds of urban elements. Although urban design is a practice process and needs much detailed and multi-dimensional description, the urban designers could only analyse urban space with script and imagination in the past, and only the urban design guideline and some drawing picture were carried out in the traditional design schema. How to improve the quality of urban space effectively and how to express the design schema perfectly are the problems troubling the urban designers for a long time. The 3D city models based spatial analysis gives the possibility of solving these problems. Compared with the traditional applications of 2D GIS in urban planning, the 3D city models based spatial analysis would be more meaningful for the urban designers. The computable analysis models could be constructed under the urban design guideline. Through the integration of analytical models and realistic visualization models, the 3D spatial analysis that is impossible to the 2D GIS can be carried out. Instead of being described with some adjective words, the designing schema and rules can be presented and analyzed in 3D virtual environment. The change of the urban fabric and the building density could be analyzed through the structure analysis. The noise environment, sunshine condition, heat environment, ventilation condition, and pollution condition could be analyzed through the physical quality analysis. The control of the height, the color and the style of the buildings, and the evaluations of the surrounding environment of the squares and the streets are also available based on the visual impact analysis. The analysis of the distribution of different function parts and its proper place are possible through function analysis of urban space. Based on the 3D city models, urban designers can express the design concept clearly. With the analysis functions, it is also possible to control and improve the quality of urban space effectually. There would be a great progress when we design the urban space with an operational 3D analytical system, like Cyber City GIS.
15 December 2021 in IEEE Transactions on Intelligent Transportation Systems
Stereo matching depth estimation for rectified image pairs is of great importance to many compute vision tasks, specifically in autonomous driving. With the flourishing of convolution neural networks, responsible depth estimation of stereo matching with artificial intelligence is the most severe challenge for autonomous driving in recent years. Previous research on end-to-end trainable stereo matching networks has usually used cascading convolution blocks with down-sampling or pooling operations to extract the unary features required for matching cost construction. Such approaches lack a reconstruction stage for increasing feature map pixel-wise alignment and strength, factors which play an important role in representing the similarity between stereo image pairs. To address this issue, in this paper, we propose the progressive fusion stereo matching network (PFSM-Net). We exploit an encoder-decoder feature extraction network architecture for multi-stage and -scale dynamic feature extraction. Moreover, we propose a group-wise concatenation method to construct the cost volume, which provides a more efficient cost volume for cost aggregation. Furthermore, we propose the use of multi-scale cost aggregation networks with a progressive fusion strategy. The aggregated cost volume is progressively fused with the multi-stage and -scale cost volume as the size of the cost volume increases. Multi-stage and -scale outputs are supervised with and learned in a coarse-to-fine manner. Experimental results demonstrate that our method outperforms previous methods on the SceneFlow, KITTI 2012, and KITTI 2015 datasets.
26 April 2022 in IEEE Transactions on Instrumentation and Measurement
Point cloud registration is an essential step for free-form blade reconstruction in industrial measurement. Nonetheless, measuring defects of the 3D acquisition system unavoidably result in noisy and incomplete point cloud data, which renders efficient and accurate registration challenging. In this paper, we propose a novel global registration method that is based on the minimum potential energy (MPE) method to address these problems. The basic strategy is that the objective function is defined as the minimum potential energy optimization function of the physical registration system. The function distributes more weight to the majority of inlier points and less weight to the noise and outliers, which essentially reduces the influence of perturbations in the mathematical formulation. We decompose the solution into a globally optimal approximation procedure and a fine registration process with the trimmed iterative closest point algorithm to boost convergence. The approximation procedure consists of two main steps. First, according to the construction of the force traction operator, we can simply compute the position of the potential energy minimum. Second, to find the MPE point, we propose a new theory that employs two flags to observe the status of the registration procedure. We demonstrate the performance of the proposed algorithm on four types of blades. The proposed method outperforms the other global methods in terms of both accuracy and noise resistance.
Parcel detection is crucial to achieving automatic sorting in intelligent logistics systems. Most parcels in logistics centers are currently detected manually, imposing low efficiency and high error rate, severely limiting logistics transportation efficiency. Therefore, there is an urgent need for automated parcel detection. However, parcels in logistics centers have challenges such as dense stacking, occlusion and background interference, making it difficult for existing methods to detect parcels accurately. To address the above problem, we developed an improved Faster R-CNN-based parcel detection model spurred by current deep-learning-based object detection trends. The proposed method first solves the false detection problem due to parcel mutual occlusion by augmenting Faster R-CNN with an edge detection branch and adding object edge loss to the loss function. Furthermore, the self-attention ROI Align module is proposed to address the problem of feature misalignment caused by the quantization rounding operation in the ROI Pooling module. The module uses an attention mechanism to filter and enhance the features and uses bilinear interpolation to calculate the feature pixel values, improving detection accuracy. The implementation of the proposed method was validated using parcel images collected in the field and the public dataset SKU110K and compared with four existing parcel detection methods. The results show that our method’s Recall, Precision, map@0.5 and Fps are 96.89%, 98.76%, 98.42% and 22.83%, respectively, which significantly improves the parcel detection accuracy.