23 October 2020 in IEEE Transactions on Pattern Analysis and Machine Intelligence
This paper focuses on communication-efficient federated learning problem, and develops a novel distributed quantized gradient approach, which is characterized by adaptive communications of the quantized gradients. Specifically, the federated learning builds upon the server-worker infrastructure, where the workers calculate local gradients and upload them to the server; then the server obtain the global gradient by aggregating all the local gradients and utilizes it to update the model parameter. The key idea to save communications from the worker to the server is to quantize gradients as well as skip less informative quantized gradient communications by reusing previous gradients. Quantizing and skipping result in ‘lazy’ worker-server communications, which justifies the term L azily A ggregated Q uantized (LAQ) gradient. Theoretically, the LAQ algorithm achieves the same linear convergence as the gradient descent in the strongly convex case, while effecting major savings in the communication in terms of transmitted bits and communication rounds . Empirically, extensive experiments using realistic data corroborate a significant communication reduction compared with state-of-the-art gradient- and stochastic gradient-based algorithms.
24 March 2024 in Proceedings of the AAAI Conference on Artificial Intelligence
Multimodal learning is susceptible to modality missing, which poses a major obstacle for its practical applications and, thus, invigorates increasing research interest. In this paper, we investigate two challenging problems: 1) when modality missing exists in the training data, how to exploit the incomplete samples while guaranteeing that they are properly supervised? 2) when the missing rates of different modalities vary, causing or exacerbating the imbalance among modalities, how to address the imbalance and ensure all modalities are well-trained. To tackle these two challenges, we first introduce the variational information bottleneck (VIB) method for the cross-modal representation learning of missing modalities, which capitalizes on the available modalities and the labels as supervision. Then, accounting for the imbalanced missing rates, we define relative advantage to quantify the advantage of each modality over others. Accordingly, a bi-level optimization problem is formulated to adaptively regulate the supervision of all modalities during training. As a whole, the proposed approach features Relative advantage aware Cross-modal representation learning (abbreviated as RedCore) for missing modalities with imbalanced missing rates. Extensive empirical results demonstrate that RedCore outperforms competing models in that it exhibits superior robustness against either large or imbalanced missing rates. The code is available at: https://github.com/sunjunaimer/RedCore.
2 December 2021 in IEEE Transactions on Cloud Computing
This paper investigates the online operation of distributed data centers equipped with energy storage. We aim to minimize their long-term operational cost by optimally distributing workload among data centers and operating energy storage. However, future spatio-temporally variant uncertainties in both workload and electricity prices have been the main impediment for a performanceguaranteed online data center operation strategy. To address this issue, we develop a fully distributed online algorithm that decouples workload distribution and storage operation across the network and time by introducing well-designed virtual queues for workload and batteries into the framework of Lyapunov optimization. Theoretically, an analytical gap between the long-term operational cost achieved by our algorithm and the theoretical optimum is provided to corroborate the desirable operation strategy. Extensive simulations using the real-world workload and electricity price data demonstrate the cost-delay tradeoff that our algorithm strikes and validate the theoretical results that we obtained.
30 July 2019 in IEEE Transactions on Cloud Computing
The huge amount of energy consumed by the data centers around the world every year motivates the cloud service providers to operate data centers in a more energy efficient way. A promising solution is to turn off the idle servers, which, however, may be turned on later, incurring a significant startup cost. The problem turns to dynamically provisioning the workload, and cutting down the energy cost which includes the power to support the running of data center and the startup cost. Different from previous studies that usually consider the worst case performance guarantee when designing online algorithms, this paper considers the average case which is more practical. We propose a simple online algorithm based on the expectation of job interval of workload, which is proven to be optimal for exponential and uniform distributions and achieves tight competitive ratio $\frac{e}{e-1}$ and $\frac{4}{3}$ respectively for them. Simulations using the synthetic data verify our theoretical analysis. Numerical results employing Google's data center workload trace demonstrate that the proposed algorithm outperforms the worst case based algorithm in terms of operation cost reduction.