21 September 2021 in Transactions of the Association for Computational Linguistics
Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and thus are too resource- hungry and computation-intensive to suit low- capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted considerable research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.
21 January 2014 in Bioinformatics
Motivation: Rapid advances in next-generation sequencing (NGS) technology have led to exponential increase in the amount of genomic information. However, NGS reads contain far more errors than data from traditional sequencing methods, and downstream genomic analysis results can be improved by correcting the errors. Unfortunately, all the previous error correction methods required a large amount of memory, making it unsuitable to process reads from large genomes with commodity computers. Results: We present a novel algorithm that produces accurate correction results with much less memory compared with previous solutions. The algorithm, named BLoom-filter-based Error correction Solution for high-throughput Sequencing reads (BLESS), uses a single minimum-sized Bloom filter, and is also able to tolerate a higher false-positive rate, thus allowing us to correct errors with a 40× memory usage reduction on average compared with previous methods. Meanwhile, BLESS can extend reads like DNA assemblers to correct errors at the end of reads. Evaluations using real and simulated reads showed that BLESS could generate more accurate results than existing solutions. After errors were corrected using BLESS, 69% of initially unaligned reads could be aligned correctly. Additionally, de novo assembly results became 50% longer with 66% fewer assembly errors. Availability and implementation: Freely available at http://sourceforge.net/p/bless-ec Contact:dchen@illinois.edu Supplementary information: Supplementary data are available at Bioinformatics online.
18 May 2020 in IEEE Transactions on Computers
Quantization has been proven to be an effective method for reducing the computing and/or storage cost of DNNs. However, the trade-off of the quantization bitwidth and final accuracy is complex and non-convex, which makes it difficult to be optimized directly. Minimizing direct quantization loss (DQL) of the coefficient data is an effective local optimization method, but previous works often neglect the accurate control of the DQL, resulting in a higher loss of accuracy. In this paper, we propose a new method, VecQ, which uses vectorized quantization loss for DNN weights, aiming to achieve the minimal quantization loss and better model accuracy. In addition, in order to speed up the proposed quantization process during the model training, we accelerate the quantization process with a parameterized probability estimation method and template-based derivation calculation. We evaluate our algorithm on MNIST, CIFAR, ImageNet, IMDB movie review and THUCNews text data sets with numerical DNN models. The results demonstrate that our proposed quantization solution is more accurate and effective than the state-of-the-art approaches yet with more flexible bitwidth support. Moreover, the evaluation of our quantized models on Saliency Object Detection (SOD) tasks maintains comparable feature extraction quality with up to 16×weight size reduction.
31 March 2021 in IEEE Design & Test
This work is an introduction and a survey for the Special Issue on Machine Intelligence at the Edge. The authors argue that workloads that were formerly performed in the cloud are increasingly moving to resource-limited edge computing systems, which raises a new set of challenges for machine learning as well as new opportunities. The topic is introduced by means of building blocks ranging from edge fundamentals to edge AI enabling methodologies as well as future trends and challenges.