30 December 2024 in Visual Intelligence
The LLaMA family, a collection of foundation language models ranging from 7B to 65B parameters, has become one of the most powerful open-source large language models (LLMs) and the popular LLM backbone of multi-modal large language models (MLLMs), widely used in computer vision and natural language understanding tasks. In particular, LLaMA3 models have recently been released and have achieved impressive performance in various domains with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-constrained scenarios, we explore LLaMA3’s capabilities when quantized to low bit-width. This exploration can potentially provide new insights and challenges for the low-bit quantization of LLaMA3 and other future LLMs, especially in addressing performance degradation issues that suffer in LLM compression. Specifically, we comprehensively evaluate the 10 existing post-training quantization and LoRA fine-tuning (LoRA-FT) methods of LLaMA3 on 1-8 bits and various datasets to reveal the low-bit quantization performance of LLaMA3. To uncover the capabilities of low-bit quantized MLLM, we assessed the performance of the LLaMA3-based LLaVA-Next-8B model under 2-4 ultra-low bits with post-training quantization methods. Our experimental results indicate that LLaMA3 still suffers from non-negligible degradation in linguistic and visual contexts, particularly under ultra-low bit widths. This highlights the significant performance gap at low bit-width that needs to be addressed in future developments. We expect that this empirical study will prove valuable in advancing future models, driving LLMs and MLLMs to achieve higher accuracy at lower bit to enhance practicality.
20 November 2020 in IEEE Access
With the emergence of decentralized cryptocurrencies such as Bitcoin, it has become very difficult for law enforcement to detect suspicious activities, identify users and obtain transaction records for criminals who utilize the pseudoanonymity provided by the cryptocurrency system. Address clustering aims to break such pseudoanonymity by linking addresses that are controlled by the same user based on the information available from the blockchain, such as transaction graphs. There are already two widely used heuristics for Bitcoin address clustering. One is based on the multiple input addresses of transactions. The other is based on one-time change addresses. By reconsidering the one-time change address-based heuristic from the perspective of address reuse, we propose a new heuristic that detects one-time change addresses by eliminating addresses that are reused later as non-change addresses. As a result, this heuristic works for transactions whose one-time change addresses cannot be identified by the previous two heuristics. The experimental results for different scales of Bitcoin transaction data show that the proposed heuristic has a 0.33% mean contribution to the ratio of address reduction in addition to the contribution of the multiple input addresses and one-time change address heuristics.
16 December 2020 in BMC Bioinformatics
Background: Although biomedical publications and literature are growing rapidly, there still lacks structured knowledge that can be easily processed by computer programs. In order to extract such knowledge from plain text and transform them into structural form, the relation extraction problem becomes an important issue. Datasets play a critical role in the development of relation extraction methods. However, existing relation extraction datasets in biomedical domain are mainly human-annotated, whose scales are usually limited due to their labor-intensive and time-consuming nature.Results: We construct BioRel, a large-scale dataset for biomedical relation extraction problem, by using Unified Medical Language System as knowledge base and Medline as corpus. We first identify mentions of entities in sentences of Medline and link them to Unified Medical Language System with Metamap. Then, we assign each sentence a relation label by using distant supervision. Finally, we adapt the state-of-the-art deep learning and statistical machine learning methods as baseline models and conduct comprehensive experiments on the BioRel dataset.Conclusions: Based on the extensive experimental results, we have shown that BioRel is a suitable large-scale datasets for biomedical relation extraction, which provides both reasonable baseline performance and many remaining challenges for both deep learning and statistical methods.
22 February 2023 in IEEE Transactions on Neural Networks and Learning Systems
Deep neural networks, such as the deep-FSMN, have been widely studied for keyword spotting (KWS) applications while suffering expensive computation and storage. Therefore, network compression technologies such as binarization are studied to deploy KWS models on edge. In this article, we present a strong yet efficient binary neural network for KWS, namely, BiFSMNv2, pushing it to the real-network accuracy performance. First, we present a dual-scale thinnable 1-bit-architecture (DTA) to recover the representation capability of the binarized computation units by dual-scale activation binarization and liberate the speedup potential from an overall architecture perspective. Second, we also construct a frequency-independent distillation (FID) scheme for KWS binarization-aware training, which distills the high-and low-frequency components independently to mitigate the information mismatch between full-precision and binarized representations. Moreover, we propose the learning propagation binarizer (LPB), a general and efficient binarizer that enables the forward and backward propagation of binary KWS networks to be continuously improved through learning. We implement and deploy BiFSMNv2 on ARMv8 real-world hardware with a novel fast bitwise computation kernel (FBCK), which is proposed to fully use registers and increase instruction throughput. Comprehensive experiments show our BiFSMNv2 outperforms the existing binary networks for KWS by convincing margins across different datasets and achieves comparable accuracy with the full-precision networks (only a tiny 1.51% drop on Speech Commands V1–12). We highlight that benefiting from the compact architecture and optimized hardware kernel, BiFSMNv2 can achieve an impressive 25.1 $\times$ speedup and 20.2 $\times$ storage-saving on edge hardware.
22 July 2020 in IEEE Access
Along with the rapidly increasing massive online data, recommender systems have been used as an effective approach for filtering useful information, which have been widely adopted in many web applications. In recent years, new techniques such as neural network and deep learning have demonstrated its effectiveness in studies of recommender systems. In this paper, we propose a novel approach for collaborative filtering with implicit feedback based on knowledge graph embedding. The basic ideal is to model the interactions between users and items as an interaction knowledge graph with a single relation, whose vector representation is learned through knowledge graph embedding. Base on the learned representation, the collaborative filtering problem is converted into link prediction in the interaction knowledge graph. KGECF, a neural network for knowledge graph embedding based collaborative filtering, is proposed based on this ideal and the RotatE knowledge graph embedding model. Experimental results on five datasets with different characteristics show that the KGECF model achieves the state-of-the-art (SOTA) performance on all datasets. And unlike other SOTA models that have obvious performance drops on AMusic and AToy datasets, our model’s performance is very stable across all datasets.