12 April 2025 in Neural Networks
29 July 2024 in IEEE Access
Large language models (LLMs) have demonstrated their vast potential and value in natural language processing tasks and beyond. However, when these models are applied to develop consultation systems for industrial domains, such as e-government, intelligent diagnosis, and legal consultancy, they encounter many unresolved technical issues. These include the vast scale of the models, lack of tight fusion with existing industry knowledge, along with occurrences of model hallucinations and inadequate explainability. Unlike general-purpose dialogue systems, building a consultation system for a specific industrial domain requires not only the integration of extensive external knowledge from the Internet but also the incorporation of precise, specialized knowledge from specific industries, making the challenges even more complex. In response to these challenges, we propose the Inter-External Knowledge Fusion Contrastive Learning Technique. This technique facilitates the integration of internal industry knowledge with widely accessible external knowledge from the Internet and provides a universal framework for building lightweight, domain-specific consultation systems. Utilizing this technique enables the straightforward creation of precise, professional, and constantly updated domain-specific consultation systems applicable across various industries. Overcoming the inherent limitations associated with LLMs, this technique achieves a performance level comparable to LLMs. To validate the effectiveness of our proposed technique, we conducted extensive experiments in the development of real-world industry consulting systems. By testing on seven real datasets covering diverse tasks, we demonstrate the system’s exceptional performance: our lightweight consultation system utilizes only 4% of the parameters of an LLM but achieves over 90% of its performance level.
30 April 2024 in IEEE Access
Cross-domain Consultation Systems have become essential in numerous critical applications, for instance, an online citizen complaint system. However, addressing complaints with distinct orality characteristics often necessitates retrieving and integrating knowledge from diverse professional domains. This scenario represents a typical cross-domain problem. Nevertheless, the prevailing approach of utilizing generative large language models to tackle this problem presents challenges including model scale and drawbacks like hallucination and limited interpretability. To address these challenges, we proposed a novel approach called the Core-View Contrastive Learning (CVCL) network. Leveraging contrastive learning techniques with an integrated core-adaptive augmentation module, the CVCL network achieves accuracy in cross-domain information matching. Our objective is to construct a lightweight, precise, and interpretable cross-domain consultation system, overcoming the limitations encountered with large language models in addressing such challenges. Empirical validation of our proposed method using real-world datasets demonstrates its effectiveness. Our experiments show that the proposed method achieves comparable performance to large language models in terms of accuracy in text-matching tasks and surpasses the best baseline model by over 24 percentage points in F1-score for classification tasks. Additionally, our lightweight model achieved a performance level of 96% compared to the full model, while utilizing only 6% of the parameters.