20 February 2017 in IEEE Transactions on Signal and Information Processing over Networks
In this paper, we propose the primal-dual method of multipliers (PDMM) for distributed optimization over a graph. In particular, we optimize a sum of convex functions defined over a graph, where every edge in the graph carries a linear equality constraint. In designing the new algorithm, an augmented primal-dual Lagrangian function is constructed which smoothly captures the graph topology. It is shown that a saddle point of the constructed function provides an optimal solution of the original problem. Further under both the synchronous and asynchronous updating schemes, PDMM has the convergence rate of O(1=K) (where K denotes the iteration index) for general closed, proper and convex functions. Other properties of PDMM such as convergence speeds versus different parametersettings and resilience to transmission failure are also investigated through the experiments of distributed averaging.
24 May 2021 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Due to lack of data, overfitting ubiquitously exists in real-world applications of deep neural networks (DNNs). We propose advanced dropout, a model-free methodology, to mitigate overfitting and improve the performance of DNNs. The advanced dropout technique applies a model-free and easily implemented distribution with parametric prior, and adaptively adjusts dropout rate. Specifically, the distribution parameters are optimized by stochastic gradient variational Bayes in order to carry out an end-to-end training. We evaluate the effectiveness of the advanced dropout against nine dropout techniques on seven computer vision datasets (five small-scale datasets and two large-scale datasets) with various base models. The advanced dropout outperforms all the referred techniques on all the datasets. We further compare the effectiveness ratios and find that advanced dropout achieves the highest one on most cases. Next, we conduct a set of analysis of dropout rate characteristics, including convergence of the adaptive dropout rate, the learned distributions of dropout masks, and a comparison with dropout rate generation without an explicit distribution. In addition, the ability of overfitting prevention is evaluated and confirmed. Finally, we extend the application of the advanced dropout to uncertainty inference, network pruning, text classification, and regression. The proposed advanced dropout is also superior to the corresponding referred methods. Codes are available at https://github.com/PRIS-CV/AdvancedDropout .
10 September 2018 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
In an active noise control (ANC) system, computational complexity is one major concern when designing practical control algorithms. For an ANC system with multiple secondary sources and error microphones, one approach to reducing computational complexity is to apply a decentralized control scheme rather than centralized approaches. A decentralized scheme attempts to control a number of small-size ANC subsystems independently. In this paper, we consider the decentralized control of a two-channel ANC system tackling a noise disturbance in the frequency domain, where each channel consists of one secondary source and one error microphone. We propose a decentralized control method that is able to achieve the same noise reduction performance as the centralized controller with guaranteed convergence. The key step in designing the control method is to properly shape the eigenvalues of a matrix that models the two-channel secondary paths for each frequency index.
10 August 2021 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Channel attention mechanisms have been commonly applied in many visual tasks for effective performance improvement. It is able to reinforce the informative channels as well as to suppress the useless channels. Recently, different channel attention modules have been proposed and implemented in various ways. Generally speaking, they are mainly based on convolution and pooling operations. In this paper, we propose Gaussian process embedded channel attention (GPCA) module and further interpret the channel attention schemes in a probabilistic way. The GPCA module intends to model the correlations among the channels, which are assumed to be captured by beta distributed variables. As the beta distribution cannot be integrated into the end-to-end training of convolutional neural networks (CNNs) with a mathematically tractable solution, we utilize an approximation of the beta distribution to solve this problem. To specify, we adapt a Sigmoid-Gaussian approximation, in which the Gaussian distributed variables are transferred into the interval [0,1]. The Gaussian process is then utilized to model the correlations among different channels. In this case, a mathematically tractable solution is derived. The GPCA module can be efficiently implemented and integrated into the end-to-end training of the CNNs. Experimental results demonstrate the promising performance of the proposed GPCA module. Codes are available at https://github.com/PRIS-CV/GPCA .
1 June 2020 in The Journal of the Acoustical Society of America
Due to their low computational complexity, reduced wiring cost, and flexibility of scaling up, decentralized multiple channel active control systems are attractive in many applications. In a decentralized multiple channel active control system, a number of small subsystems are constructed, which are updated independently with only the associated error signals. In this letter, a time domain two channel decentralized control algorithm is proposed to achieve the similar noise reduction performance as the centralized one. Auxiliary filters are introduced to filter the reference signal for control filter update and a unique design method is proposed to shape the frequency response of the auxiliary filters. The simulation results using the measured impulse responses demonstrate the efficacy of the proposed algorithm for broadband noise control.