This paper proposes the “ultimate compression” method as a solution to the expansive computation and high storage costs required by state-of-the-art neural network models in inference. Our approach uniquely combines tensor decomposition techniques with binary neural networks to create efficient deep neural network models optimized for edge inference. The process includes training floating-point models, applying tensor decomposition algorithms, binarizing the decomposed layers, and fine tuning the resulting models. We evaluated our approach in various state-of-the-art deep neural network architectures on multiple datasets, such as MNIST, CIFAR-10, CIFAR-100, and ImageNet. Our results demonstrate compression ratios of up to 169×, with only a small degradation in accuracy (1–2%) compared to binary models. We employed different optimizers for training and fine tuning, including Adam and AdamW, and used norm grad clipping to address the exploding gradient problem in decomposed binary models. A key contribution of this work is a novel layer sensitivity-based rank selection algorithm for tensor decomposition, which outperforms existing methods such as random selection and Variational Bayes Matrix Factorization (VBMF). We conducted comprehensive experiments using six different models and present a case study on crowd-counting applications, demonstrating the practical applicability of our method. The ultimate compression method outperforms binary neural networks and tensor decomposition when applied individually in terms of storage and computation costs. This positions it as one of the most effective options for deploying compact and efficient models in edge devices with limited computational resources and energy constraints.
This paper presents an innovative method for designing finite impulse response (FIR) filters. The method optimizes the desired frequency response attributes while simultaneously increasing the sparsity of the filter coefficients. Traditional FIR filter design techniques, such as the window method (FirW) and the Parks–McClellan (FirPM) algorithm, excel in meeting precise frequency-domain requirements but often result in dense impulse responses. In scenarios with limited resources, a sparse filter, which has numerous zero or nearly zero coefficients, has advantages such as decreased computational complexity, lower power consumption, and simplified hardware integration. The proposed (L1/L2 regularization) approach defines filter design as an iterative optimization challenge that decreases a composite objective function. This function combines an error term based on the L2-norm to measure deviation from the target frequency response and an L1-norm-based regularization term to encourage coefficient sparsity. By adjusting the regularization parameter λ, users can balance performance in the frequency domain with the level of impulse response sparsity. Extensive simulations reveal that compared with filters designed using FirW and FirPM, this method produces filters with competitive frequency characteristics while achieving significantly higher sparsity. This finding highlights its considerable potential for effective hardware and software implementation. The proposed FIR filter design method presents a compelling alternative to conventional paradigms, particularly for applications where implementation efficiency is a critical design constraint.
Group equivariant convolutional neural networks (G-CNNs) achieve superior sample efficiency by encoding symmetry into network architecture, yet their computational overhead (up to 3.78× slower inference and 4.63× more multiply–accumulate operations) hinders deployment on resource-constrained edge devices. Existing pruning methods cannot be applied directly: arbitrarily removing weights breaks the group representation structure and degrades equivariance. We characterize the complete design space of equivariance-preserving compression, proving that exactly two axes leave a convolutional layer equivariant: irrep-bundle pruning, which reduces irreducible-representation multiplicities, and orbit-wise pruning, which removes complete spatial orbits from kernel supports; via Schur’s lemma, no third structure-preserving axis exists. This completeness result, rather than the use of representation theory itself, is our central contribution. We turn it into practice through direct sub-filter extraction, which yields real convolutional parameter reduction (up to 83%) and 1.4–2.9× measured inference speedup, unlike masking, which gives no real speedup. Across three datasets (MNIST, CIFAR-10, EuroSAT) and three symmetry groups (C4, D4, SO(2)), compression is nearly lossless on strongly symmetric data: the 4-layer EuroSAT model drops only 1.07% at 83% reduction. On weakly symmetric data (CIFAR-10), the pruned model can even gain 2.6 points, but our analysis attributes this to relaxing a mismatched equivariance constraint rather than to pruning itself; the value of pruning over from-scratch training scales with the data’s symmetry strength.
This paper proposes a storage-efficient ensemble classification to overcome the low inference accuracy of binary neural networks (BNNs). When external power is enough in a dynamic powered system, classification results can be enhanced by aggregating outputs of multiple BNN classifiers. However, memory requirements for storing multiple classifiers are a significant burden in the lightweight system. The proposed scheme shares the filters from a trained convolutional neural network (CNN) model to reduce storage requirements in the binarized CNNs instead of adopting the fully independent classifier. While several filters are shared, the proposed method only trains unfrozen learnable parameters in the retraining step. We compare and analyze the performances of the proposed ensemble-based systems depending on various ensemble types and BNN structures on CIFAR datasets. Our experiments conclude that the proposed method using the filter sharing can be scalable with the number of classifiers and effective in enhancing classification accuracy. With binarized ResNet-20 and ReActNet-10 on the CIFAR-100 dataset, the proposed scheme can achieve 56.74% and 70.29% Top-1 accuracies with 10 BNN classifiers, which enhances performance by 7.6% and 3.6% compared with that using a single BNN classifier.
Neural scaling laws describe how model performance improves as a power law with size, but existing work has focused almost entirely on models above 100 M parameters. The regime below 20 million parameters, where TinyML and edge AI systems operate, remains largely unexamined. We train 90 models spanning 22 K to 19.8 M parameters across two architecture families (a plain ConvNet and MobileNetV2) on CIFAR-100, varying width while holding depth and training protocol fixed. Both architectures follow approximate power laws, with exponents of (ScaleCNN) and (MobileNetV2). However, the power law does not hold uniformly: local exponents decay with scale, and MobileNetV2 saturates at 19.8 M parameters (), hitting a data wall. The structure of errors also changes with scale. The Jaccard overlap between error sets of the smallest and largest ScaleCNN models is only 0.35; compression changes which inputs are misclassified, not merely how many. Small models develop a triage strategy, concentrating capacity on easy classes (Gini of per-class accuracy: 0.26 at 22 K params vs. 0.09 at 4.7 M) while effectively abandoning the hardest ones (bottom-5 class accuracy: 10% vs. 53%). The smallest models achieve the lowest ECE values (0.013 vs. peak 0.110 at mid-size), reversing the typical overconfidence–capacity relationship, though this partly reflects a global-mean matching artifact rather than well-calibrated per-bin confidence. On CIFAR-100, aggregate accuracy alone is therefore a misleading basis for edge deployment decisions; validation must happen at the target model size. All findings in this study are based on CIFAR-100 (32 × 32, 100 classes); their generalizability to other datasets, resolutions, and architectures remains to be verified.