Currently, audio-visual speech separation methods utilize the speaker's audio and visual correlation information to help separate the speech of the target speaker. However, these methods commonly use the approach of feature concatenation with linear mapping to obtain the fused audio-visual features, which prompts us to conduct a deeper exploration for audio-visual fusion. Therefore, in this paper, according to the speaker's mouth landmark movements during speech, we propose a novel time-domain single-channel audio-visual speech separation method: audio-visual fusion with temporal convolution attention network for speech separation model (AVTCA). In this method, we design temporal convolution attention network (TCANet) based on the attention mechanism to model the contextual relationships between audio and visual sequences, and use TCANet as the basic unit to construct sequence learning and fusion network. In the whole deep separation framework, we first use cross attention to focus on the cross-correlation information of the audio and visual sequences, and then we use the TCANet to fuse the audio-visual feature sequences with temporal dependencies and cross-correlations. Afterwards, the fused audio-visual features sequences will be used as input to the separation network to predict mask and separate the source of each speaker. Finally, this paper conducts comparative experiments on Vox2, GRID, LRS2 and TCD-TIMIT datasets, indicating that AVTCA outperforms other state-of-the-art (SOTA) separation methods. Furthermore, it exhibits greater efficiency in computational performance and model size.

Local binary pattern (LBP) is sensitive to noise. Noise-resistant LBP (NRLBP) addresses this problem by thresholding local neighboring pixels into three-valued states (i.e., 0, 1 and uncertain bits) and recovering uncertain bits via an error-correction mechanism. In this paper, we extend NRLBP to deal with color images and propose a robust color image descriptor called Color context binary pattern (CCBP). In CCBP, we employ scale context and neighbor context to progressively correct the encoded bits. First, we encode intra-channel local neighboring pixels into three-valued states in scale space and use majority voting to correct all states across scales. Then, we compute inter-channel color feature distances and correct the uncertain bits via neighboring bit propagation. Finally, we construct the image descriptor by concatenating all histograms based on the corrected binary codes. Experiments on four benchmark databases demonstrate the robustness of CCBP for color image classification under very low signal-to-noise ratio levels.

As a selective state space model, Mamba exhibits outstanding performance and efficiency in sequence modeling tasks. Therefore, in this paper, we use Mamba as the fundamental network component to construct a novel speech separation model, Speech Conv-Mamba. Specifically, this model embeds Mamba within a U-shaped convolutional network to build the encoder and decoder network for high-dimensional representation and waveform reconstruction of speech signals. Additionally, we stack multiple temporal dilated convolutions and Mamba to create the separation network for separation task. Our comparative experiments on the GRID2Mix and Libri2Mix datasets demonstrate that the proposed model Speech Conv-Mamba, which achieves 98% and 89% of SepFormer's separation accuracy on two datasets using only 9% (2.4 M) of its model size, provides much less computational complexity and training cost.