Violence detection in videos can help maintain public order, detect crimes, or provide timely assistance. In this paper, we aim to leverage multimodal information to determine whether successive frames contain violence. Specifically, we propose an audiovisual dependency attention (AVD-attention) module modified from the co-attention architecture to fuse visual and audio information, unlike commonly used methods such as the feature concatenation, addition, and score fusion. Because the AVD-attention module’s dependency map contains sufficient fusion information, we argue that it should be applied more sufficiently. A combination pooling method is utilized to convert the dependency map to an attention vector, which can be considered a new feature that includes fusion information or a mask of the attention feature map. Since some information in the input feature might be lost after processing by attention modules, we employ a multimodal low-rank bilinear method that considers all pairwise interactions among two features in each time step to complement the original information for output features of the module. AVD-attention outperformed co-attention in experiments on the XD-Violence dataset. Our system outperforms state-of-the-art systems.

Few-shot Class-incremental Audio Classification (FCAC) is a task to continuously identify incremental classes with only few training samples after training the model on base classes with abundant samples. The key to solving the FCAC problem is to ensure that the model has good stability (without forgetting base classes) and strong plasticity (without overfitting incremental classes). In this paper, we propose a FCAC method which is able to adaptively mitigate the model's forgetting of base classes and overfitting of incremental classes. Our model consists of an embedding extractor and an expandable classifier. The former is the backbone of a residual network and is frozen after being trained using sufficient samples of base classes, whereas the latter can be expandable and is updated using few training samples of incremental classes in each incremental session. The expandable classifier consists of two branches and one fusion module. The two branches are designed to mitigate the model's forgetting of base classes and overfitting of incremental classes, respectively. The fusion module is designed to adaptively fuse predictions output by the above two branches. In addition, we define two losses for model training in the base and incremental sessions, respectively. Three experimental datasets (NSynth-100, FSC-89 and LS-100) are created by randomly choosing samples from audio corpora of NSynth, FSD-MIX-CLIP and LibriSpeech, respectively. Experimental results demonstrate that our proposed method outperforms all previous methods in accuracy and has advantage over most previous methods in computational load.

Label Smoothing Regularization (LSR) is a widely used tool to generalize classification models by replacing the one-hot ground truth with smoothed labels. Recent research on LSR has increasingly focused on the correlation between the LSR and Knowledge Distillation (KD), which transfers the knowledge from a teacher model to a lightweight student model by penalizing their output’s Kullback–Leibler-divergence. Based on this observation, a Teacher-free Knowledge Distillation (Tf-KD) method was proposed in previous work. Instead of a real teacher model, a handcrafted distribution similar to LSR was used to guide the student learning. Tf-KD is a promising substitute for LSR except for its hard-to-tune and model-dependent hyperparameters. This paper develops a new teacher-free framework LSR-OS-TC, which decomposes the Tf-KD method into two components: model Output Smoothing (OS) and Teacher Correction (TC). Firstly, the LSR-OS extends the LSR method to the KD regime and applies a softer temperature to the model output softmax layer. Output smoothing is critical for stabilizing the KD hyperparameters among different models. Secondly, in the TC part, a larger proportion is assigned to the uniform distribution teacher’s right class to provide a more informative teacher. The two-component method was evaluated exhaustively on the image (dataset CIFAR-100, CIFAR-10, and CINIC-10) and audio (dataset GTZAN) classification tasks. The results showed that LSR-OS can improve LSR performance independently with no extra computational cost, especially on several deep neural networks where LSR is ineffective. The further training boost by the TC component showed the effectiveness of our two-component strategy. Overall, LSR-OS-TC is a practical substitution of LSR that can be tuned on one model and directly applied to other models compared to the original Tf-KD method.

Liveness check through measuring synchrony between the lip and the voice in video sequence has been proved to be a feasible way to prevent spoofing attacks. The generic methods utilize degree of audiovisual correlation to judge consistency. However, little attention is paid to the audiovisual time drift difference between live recordings and fraudulent attacks. In this paper, the time shift difference and the delay range is concluded by delay estimation experiments on VidTIMID. According to the conclusion, we propose an improved ¿liveness¿ scores evaluation algorithm for audio-video identification system. Experimental results on the same database show that the proposed algorithm got better performance. In comparison with the traditional methods, the EER (Equal Error Rate) averagely decreases about 5%.