In unstructured environments, robots are likely to encounter desktop objects that they have never seen before. Classifying these objects precisely is a prerequisite for accomplishing object-specific manipulation tasks. However, it is time-consuming to collect large-scale object classification datasets. Inspired by the prompt tuning methods, we propose the PRO-CLIP network, which is a category measurement method for desktop objects. Specifically, PRO-CLIP performs few-shot classification based on the knowledge from pretrained vision-language model. It utilizes token-level and prompt-level optimal transportations to jointly fine-tune the learnable vision-language prompts. For token-level stage, we propose the image patch reweighting mechanism to make alignments focus on the image patches that are close to the patch prototypes. This allows the patch embeddings have converging category representations, which reduces intra-class differences of visual features. For prompt-level stage, we propose a cascading optimal transportation module to simultaneously consider task-irrelevant knowledge in zero-shot features and task-specific knowledge in few-shot features. Due to the generalization performance of task-irrelevant knowledge, the proposed module achieves feature regularization during optimal transportation. Finally, we propose the UP loss to supervise the whole network. It contains unbalanced logit-level consistency losses and visual prototype loss. The logit-level consistency losses are used to make learnable features close to zero-shot features. The prototype loss makes the visual features approach to the corresponding prototypes in distance. We demonstrate the effectiveness of our method by performing few-shot classification experiments on different datasets including desktop objects. The relevant code will be available at https://github.com/NeuCV-IRMI/proclip.
13 March 2025 in IEEE Transactions on Instrumentation and Measurement
6-DoF grasp pose detection in clutter is an essential but challenging task in the vision measurement. However, most current detection methods often fail to consider the geometric information and partial occlusions of objects in clutter, resulting in inaccurate robotic grasping and potential collisions. In this work, we propose an end-to-end grasping system to directly generate 6-DoF grasp poses from point clouds in clutter. Our system first selects the most feasible positions as the grasp centers, and then determines their grasp parameters, such as approaching vectors and in-plane rotations. Specifically, we design an optimality criterion based on the geometric information of objects to extract rich object features from the scene and identify high-graspablity areas. To alleviate the collision problem caused by occlusions, we focus on the spatial relations among objects and explicitly train a collision predictor to directly select collision-free points as grasp centers from high-graspablity areas. Based on these grasp centers, our model can easily generate diverse sets of collision- free and robust grasp poses for successful grasping. To efficiently determine grasp parameters, we exploit neighbor search to capture the local context around each grasp center and then implicitly learn relations among objects using the self-attention mechanism. Compared with previous methods, our grasping system improves performance by a large margin (>10%), and achieves state-of-the-art results on GraspNet-1Billion, a largescale dataset. Besides, our method has a high success rate for grasping in simulation and real-world experiments, verifying the validity in clutter.
9 September 2025 in IEEE Transactions on Multimedia
Gait recognition aims to identify individuals based on their body shape and walking patterns. Though much progress has been achieved driven by deep learning, gait recognition in real-world surveillance scenarios remains quite challenging to current methods. Conventional approaches, which rely on periodic gait cycles and controlled environments, struggle with the non-periodic and occluded silhouette sequences encountered in the wild. In this paper, we propose a novel framework, TrackletGait, designed to address these challenges in the wild. We propose Random Tracklet Sampling, a generalization of existing sampling methods, which strikes a balance between robustness and representation in capturing diverse walking patterns. Next, we introduce Haar Wavelet-based Downsampling to preserve information during spatial downsampling. Finally, we present a Hardness Exclusion Triplet Loss, designed to exclude low-quality silhouettes by discarding hard triplet samples. TrackletGait achieves state-of-the-art results, with 77.8% and 80.4% rank-1 accuracy on the Gait3D and GREW datasets, respectively, while using only 10.3M backbone parameters. Extensive experiments are also conducted to further investigate the factors affecting gait recognition in the wild.
2 December 2024 in IEEE Transactions on Consumer Electronics
Precisely locating the edges of objects camouflaged as background is an exceedingly complicated task. Existing camouflaged object detection (COD) methods can accurately pinpoint the positions of objects, but the edges remain ambiguous. To tackle this challenge, we propose a multi-stage refinement framework to locate the precise edges of camouflaged objects coarse-to-fine, coined SAE, which integrates “shrink" and “expose" in each stage. In this paper, we transform the process by which humans capture object edges into a multi-stage shrinking process. Firstly, we obtain the edge probability distribution by subtracting a negative prediction result from a positive prediction result, thus proposing the positive-negative edge shrink module. The edge probability distribution from the previous stage is then explicitly propagated to the current stage to enhance the edge texture details. Second, we introduce edge exposure attention to expose significant edges, which also reflects the impact of the previous stage’s edge state and edge probability distribution on the current stage’s edge state. Finally, to eliminate state transition and edge deformation anomalies, we propose the abnormal localization restriction to ensure the proper execution of “shrink” and “expose”. Experimental results show that our model significantly outperforms recent SOTA methods on three COD benchmarks.