Seyedarmin Azizi

Seyedarmin Azizi

orcid

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
5 August 2024
Verified
Conference Paper
Open access
MSMohammad Erfan Sadeghi
Mohammad Erfan Sadeghi
AFArash Fayyazi
Arash Fayyazi
SASeyedarmin Azizi
Seyedarmin Azizi
MPMassoud Pedram
Massoud Pedram
The deployment of Vision Transformers (ViTs) on hardware platforms, specially Field-Programmable Gate Arrays (FPGAs), presents many challenges, which are mainly due to the substantial computational and power requirements of their non-linear functions, notably layer normalization, softmax, and Gaussian Error Linear Unit (GELU). These critical functions pose significant obstacles to efficient hardware implementation due to their complex mathematical operations and the inherent resource count and architectural limitations of FPGAs. PEANO-ViT offers a novel approach to streamlining the implementation of the layer normalization layer by introducing a division-free technique that simultaneously approximates the division and square root function. Additionally, PEANO-ViT provides a multi-scale division strategy to eliminate division operations in the softmax layer, aided by a Padé-based approximation for the exponential function. Finally, PEANO-ViT introduces a piece-wise linear approximation for the GELU function, carefully designed to bypass the computationally intensive operations associated with GELU. In our comprehensive evaluations, PEANO-ViT exhibits minimal accuracy degradation ≤ 0.5% for DeiT-B) while significantly enhancing power efficiency, achieving improvements of 1.91X, 1.39X, and 8.01X for layer normalization, softmax, and GELU, respectively. This improvement is achieved through substantial reductions in DSP, LUT, and register counts for these non-linear operations. Consequently, PEANO-ViT enables efficient deployment of Vision Transformers on resource- and power-constrained FPGA platforms.

1 January 2024
Verified
Conference Paper
Open access
PDF file
SASeyedarmin Azizi
Seyedarmin Azizi
SKSouvik Kundu
Souvik Kundu
MPMassoud Pedram
Massoud Pedram
Seyedarmin Azizi, Souvik Kundu, Massoud Pedram. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.

20 June 2024 in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
Verified
Journal Article
Closed access
SASeyedarmin Azizi
Seyedarmin Azizi
MNMahdi Nazemi
Mahdi Nazemi
MKMehdi Kamal
Mehdi Kamal
MPMassoud Pedram
Massoud Pedram
This article presents a mixed-computation neural network processing approach for edge applications that incorporates low-precision (low-width) Posit and low-precision fixed point (FixP) number systems. This mixed-computation approach uses 4-bit Posit (Posit4), which has higher precision around 0, for representing weights with high sensitivity, while it uses 4-bit FixP (FixP4) for representing other weights. A heuristic for analyzing the importance and the quantization error of the weights is presented to assign the proper number system to different weights. In addition, a gradient approximation for Posit representation is introduced to improve the quality of weight updates in the backpropagation process. Due to the high energy consumption of the fully Posit-based computations, neural network operations are carried out in FixP or Posit/FixP. An efficient hardware implementation of an MAC operation with a first Posit operand and FixP for a second operand and accumulator is presented. The efficacy of the proposed low-precision mixed-computation approach is extensively assessed on vision and language models. The results show that on average, the accuracy of the mixed-computation is about 1.5% higher than that of FixP with a cost of 0.19% energy overhead.


8 February 2024 in arXiv
Verified
Preprint
Open access
PDF file
SASeyedarmin Azizi
Seyedarmin Azizi
MNMahdi Nazemi
Mahdi Nazemi
MPMassoud Pedram
Massoud Pedram
As Vision Transformers (ViTs) increasingly set new benchmarks in computer vision, their practical deployment on inference engines is often hindered by their significant memory bandwidth and (on-chip) memory footprint requirements. This paper addresses this memory limitation by introducing an activation-aware model compression methodology that uses selective low-rank weight tensor approximations of different layers to reduce the parameter count of ViTs. The key idea is to decompose the weight tensors into a sum of two parameter-efficient tensors while minimizing the error between the product of the input activations with the original weight tensor and the product of the input activations with the approximate tensor sum. This approximation is further refined by adopting an efficient layer-wise error compensation technique that uses the gradient of the layer's output loss. The combination of these techniques achieves excellent results while it avoids being trapped in a shallow local minimum early in the optimization process and strikes a good balance between the model compression and output accuracy. Notably, the presented method significantly reduces the parameter count of DeiT-B by 60% with less than 1% accuracy drop on the ImageNet dataset, overcoming the usual accuracy degradation seen in low-rank approximations. In addition to this, the presented compression technique can compress large DeiT/ViT models to have about the same model size as smaller DeiT/ViT variants while yielding up to 1.8% accuracy gain. These results highlight the efficacy of our approach, presenting a viable solution for embedding ViTs in memory-constrained environments without compromising their performance.

15 October 2026 in Preprints.org
Verified
Preprint
Open access
PDF file
SASeyedarmin Azizi
Seyedarmin Azizi
EPErfan Baghaei Potraghloo
Erfan Baghaei Potraghloo
MPMassoud Pedram
Massoud Pedram
Looped language models apply one shared block of Transformer layers repeatedly to the same tokens. Their effective depth, the number of layer applications per token, can grow without adding parameters and can be chosen at inference time. Recurrent depth is thus a way to scale test-time compute in latent space instead of in generated tokens. The idea appears under many names, including looped Transformers, LoopLMs, recurrent-depth models, Universal Transformers, Mixture-of-Recursions, and recursive reasoning models such as HRM and TRM. In two years it has moved from synthetic tasks to open models such as Huginn and Ouro and to pretraining on up to 28 trillion tokens. According to press reports, it is also used in a frontier-lab model. Its literature has split across architecture, training, systems, theory and safety. We present the first survey dedicated to looped language models, built on a systematic corpus of 557 core papers screened from 7,323 candidates. We introduce a unified prelude–core–coda formalism that makes explicit how parameters, compute, memory and depth are counted. We organize the field from design, training and scaling to inference systems, theory, mechanisms, capabilities, other modalities and safety. A meta-analysis classifies 822 verified empirical claims by comparison regime. It shows that most apparent contradictions about whether “a recurrence is worth a layer” come from comparing parameter-matched with compute-matched results. The evidence fits one reading, in which a recurrence is worth less than a fresh block of parameters. Looping therefore helps when parameters are the limiting resource and hurts when it replaces them at fixed compute. We derive a research agenda from measured gaps, including the absence of work on robustness, alignment and privacy. We release the corpus, the extraction records with verbatim evidence for every number, and the claims ledger.