Amir Gholami

Amir Gholami

Auto-generated profile, based on publicly available information

Amir Gholami

University of California Berkeley

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
25 March 2024 in IEEE Micro
Verified
Journal Article
Closed access
AGAmir Gholami
Amir Gholami
ZYZhewei Yao
Zhewei Yao
SKSehoon Kim
Sehoon Kim
CHColeman Hooper
Coleman Hooper
MMMichael W. Mahoney
Michael W. Mahoney
KKKurt Keutzer
Kurt Keutzer
The availability of unprecedented unsupervised training data, along with neural scaling laws, has resulted in an unprecedented surge in model size and compute requirements for serving/training LLMs. However, the main performance bottleneck is increasingly shifting to memory bandwidth. Over the past 20 years, peak server hardware FLOPS has been scaling at 3.0×/2yrs, outpacing the growth of DRAM and interconnect bandwidth, which have only scaled at 1.6 and 1.4 times every 2 years, respectively. This disparity has made memory, rather than compute, the primary bottleneck in AI applications, particularly in serving. Here, we analyze encoder and decoder Transformer models and show how memory bandwidth can become the dominant bottleneck for decoder models. We argue for a redesign in model architecture, training, and deployment strategies to overcome this memory limitation.

1 January 2024
Verified
Conference Paper
Open access
PDF file
NLNicholas Lee
Nicholas Lee
TWThanakul Wattanawong
Thanakul Wattanawong
SKSehoon Kim
Sehoon Kim
KMKarttikeya Mangalam
Karttikeya Mangalam
SSSheng Shen
Sheng Shen
AGAmir Gholami
Amir Gholami
Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael Mahoney, Kurt Keutzer, Amir Gholami. Findings of the Association for Computational Linguistics ACL 2024. 2024.

24 May 2019 in Journal of Mathematical Biology
Verified
Journal Article
Closed access
PDF file
SSShashank Subramanian
Shashank Subramanian
AGAmir Gholami
Amir Gholami
GBGeorge Biros
George Biros

7 January 2019 in Computer Methods in Applied Mechanics and Engineering
Verified
Journal Article
Closed access
KSKlaudius Scheufele
Klaudius Scheufele
AMAndreas Mang
Andreas Mang
AGAmir Gholami
Amir Gholami
CDChristos Davatzikos
Christos Davatzikos
GBGeorge Biros
George Biros
MMMiriam Mehl
Miriam Mehl

21 June 2021 in arXiv
Verified
Preprint
Open access
PDF file
AGAmir Gholami
Amir Gholami
SKSehoon Kim
Sehoon Kim
ZDZhen Dong
Zhen Dong
ZYZhewei Yao
Zhewei Yao
MMMichael W. Mahoney
Michael W. Mahoney
KKKurt Keutzer
Kurt Keutzer
As soon as abstract mathematical computations were adapted to computation on digital computers, the problem of efficient representation, manipulation, and communication of the numerical values in those computations arose. Strongly related to the problem of numerical representation is the problem of quantization: in what manner should a set of continuous real-valued numbers be distributed over a fixed discrete set of numbers to minimize the number of bits required and also to maximize the accuracy of the attendant computations? This perennial problem of quantization is particularly relevant whenever memory and/or computational resources are severely restricted, and it has come to the forefront in recent years due to the remarkable performance of Neural Network models in computer vision, natural language processing, and related areas. Moving from floating-point representations to low-precision fixed integer values represented in four bits or less holds the potential to reduce the memory footprint and latency by a factor of 16x; and, in fact, reductions of 4x to 8x are often realized in practice in these applications. Thus, it is not surprising that quantization has emerged recently as an important and very active sub-area of research in the efficient implementation of computations associated with Neural Networks. In this article, we survey approaches to the problem of quantizing the numerical values in deep Neural Network computations, covering the advantages/disadvantages of current methods. With this survey and its organization, we hope to have presented a useful snapshot of the current research in quantization for Neural Networks and to have given an intelligent organization to ease the evaluation of future research in this area.

11 November 2021 in arXiv
Verified
Preprint
Open access
PDF file
AKAditi S. Krishnapriyan
Aditi S. Krishnapriyan
AGAmir Gholami
Amir Gholami
SZShandian Zhe
Shandian Zhe
RKRobert M. Kirby
Robert M. Kirby
MMMichael W. Mahoney
Michael W. Mahoney
Recent work in scientific machine learning has developed so-called physics-informed neural network (PINN) models. The typical approach is to incorporate physical domain knowledge as soft constraints on an empirical loss function and use existing machine learning methodologies to train the model. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena for even slightly more complex problems. In particular, we analyze several distinct situations of widespread physical interest, including learning differential equations with convection, reaction, and diffusion operators. We provide evidence that the soft regularization in PINNs, which involves PDE-based differential operators, can introduce a number of subtle problems, including making the problem more ill-conditioned. Importantly, we show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN's setup makes the loss landscape very hard to optimize. We then describe two promising solutions to address these failure modes. The first approach is to use curriculum regularization, where the PINN's loss term starts from a simple PDE regularization, and becomes progressively more complex as the NN gets trained. The second approach is to pose the problem as a sequence-to-sequence learning task, rather than learning to predict the entire space-time at once. Extensive testing shows that we can achieve up to 1-2 orders of magnitude lower error with these methods as compared to regular PINN training.