16 April 2024 in IEEE Signal Processing Letters
The weakly-supervised audio-visual video parsing (AVVP) task aims toparse a video into temporal events and predict their modality-specific categories. Current works primarily focus on refining training strategies and follow the framework fusing signals only at the segment level. However, they miss the point that video events, being composed of consecutive segments, require the integration of both local and global contexts to fully capture their essence. In this letter, we present the L ocal- G lobal F usion Net work ( LGFNet ), designed to facilitate multi-level interaction between audio and visual signals. Specifically, we create a two-dimensional map to generate multi-scale event proposals for both audio and visual modalities. Subsequently, we fuse audio and visual signals at both segment and event levels with a novel boundary-aware feature aggregation method, enabling the simultaneous capture of local and global information. To enhance the temporal alignment between the two modalities, we employ segment-level and event-level contrastive learning. In-depth experiments demonstrate the superiority of our LGFNet.
16 April 2024 in IEEE Signal Processing Letters
The robustness to spatio-temporal sampling is significant for point cloud video understanding. Previous works overlook this issue and usually suffer notable performance drops when point densities and frame rates are changed. To remedy this, we propose a point spatio-temporal pyramid (PoST-Py) to improve the sampling robustness of point cloud video modeling. Specifically, we propose a pluggable PoST-Py to collect multi-scale feature maps from different layers of the backbone. Then, these features are integrated into a unified representation. This allows the model to capture multi-scale spatio-temporal information simultaneously. In addition, we employ the temporal cardinality difference to enhance the features to capture motion information. Extensive experiments show that PoST-Py achieves state-of-the-art performance, particularly with a notable improvement of over 2% under varying point sampling. This demonstrates the improved robustness of our method.