Due to industrial demands to handle increasing amounts of training data, lower the cost of computing one model at a time, and lessen the ecological effects of intensive computing resource consumption, the job of speeding the training of deep neural networks becomes exceedingly challenging. Adaptive Online Importance Sampling and IDS are two brand-new methods for accelerating training that are presented in this research. On the one hand, Adaptive Online Importance Sampling accelerates neural network training by lowering the number of forward and backward steps depending on how poorly a model can identify a given data sample. On the other hand, Intellectual Data Selection accelerates training by removing semantic redundancies from the training dataset and subsequently lowering the number of training steps. The study reports average 1.9x training acceleration for ResNet50, ResNet18, MobileNet v2 and YOLO v5 on a variety of datasets: CIFAR-100, CIFAR-10, ImageNet 2012 and MS COCO 2017, where training data are reduced by up to five times. Application of Adaptive Online Importance Sampling to ResNet50 training on ImageNet 2012 results in 2.37 times quicker convergence to 71.7% top-1 accuracy, which is within 5% of the baseline. Total training time for the same number of epochs as the baseline is reduced by 1.82 times, with an accuracy drop of 2.45 p.p. The amount of time required to apply Intellectual Data Selection to ResNet50 training on ImageNet 2012 is decreased by 1.27 times with a corresponding decline in accuracy of 1.12 p.p. Applying both methods to ResNet50 training on ImageNet 2012 results in 2.31 speedup with an accuracy drop of 3.5 p.p.

As modern neural network training and fine-tuning requires a lot of computational resources, there is a huge demand for novel, specialized algorithms for efficient and cost-effective training procedures. Aggressive Loss-based Elimination of Samples (ALOE) is an innovative method that operates with training samples based on losses obtained from a currently trained model or a pre-trained one. ALOE is designed to accelerate the fine-tuning process of Large Language Models and is perfectly integrated with the state-of-the-art Parameter-Efficient Fine-Tuning method LoRA. ALOE is a two-stage fine-tuning acceleration method. The two stages of ALOE are called offline and online. The proposed method is based on the idea that reducing the number of samples due to a certain rule decreases the number of training steps, thus reducing the overall fine-tuning time for LLM. This reduction allows to either get a fine-tuned version of the model faster or to perform more training iterations within the same time period as the fine-tuning baseline (ALOE Max). The ALOE (Offline) performs dataset reduction before the fine-tuning starts, while the ALOE (Online) selects the training samples from each training batch during the fine-tuning process. Results demonstrate significant acceleration by 45.6% in average across 6 models: GPT-2 S, GPT-2 M, DeBERTa-V2-XL, LLaMA-7B, LLaMA-2-7B, LLaMA-2-13B with average accuracy improvement by 5.91% in comparison to the fine-tuning results obtained with the use of LoRA method. ALOE (Offline) is able to accelerate GPT-2 M E2E-NLG fine-tuning by up to 92% with 1.2% BLEU improvement.

Modern Multimodal Large Language Models have increased demands on computational resources required for both pretraining and fine-tuning procedures. This challenge is primarily attributed to the backpropagation step because the computation of gradients is time-consuming and memory-intensive. This paper aims to alleviate the presented issues, and introduces novel fine-tuning strategy. Low-Rank Adaptation with Hebb Rapid Optimization (LoRA-HeRO) effectively combines the gradient-based method of LoRA fine-tuning with a local learning rule. An extra feature of the proposed algorithm is weight importance analysis, that identifies Transformer blocks for vanilla LoRA update. Additionally, it is possible to perform the analysis of model convergence during the fine-tuning process. LoRA-HeRO achieves lossless fine-tuning acceleration for InternVL-1B model by up to 48% and StableDiffusionV1-4 fine-tuning acceleration by 50% compared to conventional LoRA fine-tuning.

Large Language Models (LLMs) have become central advancements in artificial intelligence, particularly in machine learning, natural language processing, and computer vision. Their ability to understand and generate human-like text has made them crucial in applications ranging from automated translation to text generation. Despite the vast capabilities of pre-trained LLMs, their deployment in specialized domains often requires fine-tuning, an adaptation process constrained by high resource demands and extensive computational time. At present, the most prominent approach for fine-tuning acceleration is LoRA, which involves inserting into the model trainable low-rank adapters while freezing the rest of the parameters. However, this state-of-the-art approach is significantly limited by the requirement to manually attach adapters to each Transformer block, leading to computational overhead. Developing novel fine-tuning strategies that overcome this limitation presents significant opportunities for reducing fine-tuning time without degrading the quality. This paper addresses this challenge by introducing an innovative fine-tuning solution that dynamically assigns LoRA adapters to Transformer blocks of the model based on the importance and convergence status, significantly enhancing the efficiency of the process. Our method improves upon existing techniques, providing a considerable fine-tuning acceleration on average of 55% without quality drop compared to LoRA.