14 February 2025 in IEEE Access
The advent of Large Language Models (LLMs) has revolutionized Natural Language Processing (NLP), offering unprecedented capabilities for understanding and generating human-like texts. However, the effectiveness of these models in specific domains often requires fine-tuning, which can be computationally expensive and increases the risk of overfitting. To address these challenges, we introduce Task-Agnostic Adaptive Activation Scaling Network (TAAS-Net), a novel multi-task learning model architecture designed to adapt pre-trained LLMs to new tasks with minimal computational and storage overhead. TAAS-Net leverages an adaptive activation scaling network to enhance parameter efficiency and prevent overfitting, thereby improving model generalization. It incorporates a task-agnostic adaptive activation scaling layer that introduces task-specific, learnable activation scaling vectors, enhancing the model’s adaptability across various tasks. In addition, Low-Rank Adaptation within the attention modules facilitates the transfer of knowledge across different tasks. TAAS-Net’s integration of multi-task learning and parameter-efficient fine-tuning schemes enables efficient fine-tuning in multi-task scenarios, with a reduced parameter count compared to existing methods such as MoE-LoRA and X-LoRA. Our model also handles mixed-task batches during inference, thereby enhancing versatility and operational efficiency. Through a series of experiments on benchmark datasets, including the General Language Understanding Evaluation (GLUE), IMDB, and Agnews, TAAS-Net demonstrated superior performance over existing baselines, underscoring its potential for applications requiring rapid deployment and high accuracy. The architectural design of TAAS-Net streamlines the model adaptation process and reduces resource demands, offering a scalable and resource-efficient solution for adapting LLMs to new tasks.