Yingtao Zhang

Dr. Yingtao Zhang


Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
27 May 2024 in Preprints.org
Verified
Preprint
Open access
PDF file
YZYingtao Zhang
Yingtao Zhang
HBHaoli Bai
Haoli Bai
HLHaokun Lin
Haokun Lin
JZJialin Zhao
Jialin Zhao
LHLu Hou
Lu Hou
Carlo Vittorio CannistraciCarlo Vittorio Cannistraci
Carlo Vittorio Cannistraci
With the rapid growth of large language models (LLMs), there is increasing demand for memory and computation in LLMs. Recent efforts on post-training pruning of LLMs aim to reduce the model size and computation requirements, yet the performance is still sub-optimal. In this paper, we present a plug-and-play solution for post-training pruning of LLMs. The proposed solution has two innovative components: 1) Relative Importance and Activations (RIA), a new pruning metric that jointly considers the weight and activations efficiently on LLMs; and 2) Channel Permutation, a new approach to maximally preserve important weights under N:M sparsity. The two proposed components can be readily combined to further enhance the N:M semi-structured pruning of LLMs. Our empirical experiments show that RIA alone can already surpass all existing post-training pruning methods on prevalent LLMs, e.g., LLaMA ranging from 7B to 65B. Furthermore, N:M semi-structured pruning with channel permutation can even outperform the original LLaMA2-70B on zero-shot tasks, together with practical speed-up on specific hardware. Our code is available at: https://github.com/biomedical-cybernetics/Relative-importance-and-activation-pruning

27 May 2024 in Preprints.org
Verified
Preprint
Open access
PDF file
YZYingtao Zhang
Yingtao Zhang
JZJialin Zhao
Jialin Zhao
WWWenjing Wu
Wenjing Wu
AMAlessandro Muscoloni
Alessandro Muscoloni
Carlo Vittorio CannistraciCarlo Vittorio Cannistraci
Carlo Vittorio Cannistraci
Sparse training (ST) aims to ameliorate deep learning by replacing fully connected artificial neural networks (ANNs) with sparse or ultra-sparse ones, such as brain networks are, therefore it might benefit to borrow brain-inspired learning paradigms from complex network intelligence theory. Here, we launch the ultra-sparse advantage challenge, whose goal is to offer evidence on the extent to which ultra-sparse (around 1% connection retained) topologies can achieve any leaning advantage against fully connected. Epitopological learning is a field of network science and complex network intelligence that studies how to implement learning on complex networks by changing the shape of their connectivity structure (epitopological plasticity). One way to implement Epitopological (epi- means new) Learning is via link prediction: predicting the likelihood of non-observed links to appear in the network. Cannistraci-Hebb learning theory inspired the CH3-L3 network automata rule for link prediction which is effective for general-purpose link prediction. Here, starting from CH3-L3 we propose Epitopological Sparse Meta-deep Learning (ESML) to apply Epitopological Learning to sparse training. In empirical experiments, we find that ESML learns ANNs with ultra-sparse hyperbolic (epi-)topology in which emerges a community layer organization that is meta-deep (meaning that each layer also has an internal depth due to power-law node hierarchy). Furthermore, we discover that ESML can in many cases automatically sparse the neurons during training (arriving even to 30% neurons left in hidden layers), this process of node dynamic removal is called percolation. Starting from this network science evidence, we design Cannistraci-Hebb training (CHT), a 4-step training methodology that puts ESML at its heart. We conduct experiments on 7 datasets and 5 network structures comparing CHT to dynamic sparse training SOTA algorithms and the fully connected counterparts. The results indicate that, with a mere 1% of links retained during training, CHT surpasses fully connected networks on VGG16, GoogLeNet, ResNet50, and ResNet152. This key finding is an evidence for ultra-sparse advantage and signs a milestone in deep learning. CHT acts akin to a gradient-free oracle that adopts CH3-L3-based epitopological learning to guide the placement of new links in the ultra-sparse network topology to facilitate sparse-weight gradient learning, and this in turn reduces the convergence time of ultra-sparse training. Finally, CHT offers the first examples of parsimony dynamic sparse training because, in many datasets, it can retain network performance by percolating and significantly reducing the node network size. Our code is available at: https://github.com/biomedical-cybernetics/Cannistraci-Hebb-training.

30 September 2025 in Preprints.org
Verified
Preprint
Open access
PDF file
YZYingtao Zhang
Yingtao Zhang
WGWenqi Gu
Wenqi Gu
WHWen Hu
Wen Hu
JLJianguo Li
Jianguo Li
Carlo Vittorio CannistraciCarlo Vittorio Cannistraci
Carlo Vittorio Cannistraci
Transformers are the backbone of state-of-the-art systems across language, vision, and multimodal learning tasks, yet the relevance scale of their functional blocks (self-attention and feed-forward networks) is typically constant across inputs and depth. This static design neglects context-sensitive regulation of information flow through residual pathways. We introduce the \emph{contextual modulator}: a lightweight, input-aware mechanism that can scale the outputs of linear sublayers within a block or the entire block output at token- and channel-level granularity. The modulator is implemented via compact parametric functions and adds negligible parameter overhead. Building on this idea, we propose Transponder, which integrates contextual modulators throughout Transformer blocks to endow functional residual architectures with fine-grained, input-adaptive control. Transponder provides evident improvement over six other scaling or normalization methods across LLaMA backbones ranging from 60M to 250M parameters, yielding consistent perplexity reductions with $<1\%$ additional parameters. Analysis reveals depth-, module-, and token-specific scaling patterns, indicating that learned modulators act as input-adaptive regulators of residual information flow. Transponder provides a simple, general mechanism to augment Transformer-based models with context-sensitive modulators, providing robust and significant performance improvements without substantial architectural changes.

6 May 2026 in Preprints.org
Verified
Preprint
Open access
PDF file
WGWenqi Gu
Wenqi Gu
YZYingtao Zhang
Yingtao Zhang
AMAlessandro Muscoloni
Alessandro Muscoloni
Carlo Vittorio CannistraciCarlo Vittorio Cannistraci
Carlo Vittorio Cannistraci
Logistic and logit functions play important roles in modern science, serving as foundational tools in various applications including artificial neural network (ANN). While there are functions that could produce distinct logistic and logit curves, no single, unified framework has been developed to generate both logistic and logit curves. We introduce a generalized logistic–logit function (CMG-GLLF) to fill this gap. CMG-GLLF provides four interpretable and trainable parameters that allow explicit control over: curve type and steepness, asymmetry, upper and lower limits of x- and y-axes. CMG-GLLF’s potential is explored in basic machine intelligence tasks. As a proof-of-concept on how this function can improve performance of deep learning, we propose a trainable input feature modulator (IFM) that consists in learning the parameters of the CMG-GLLF for each input layer node during backpropagation for multi-layer perceptron (MLP), which is a fundamental building block of many complex network architectures. Compared to various other learnable functions, across 3 different optimizers, CMG-GLLF allows superior MLP’s accuracy and stable training behavior on CIFAR-10 and CIFAR-100 image classification, but at the cost of increased computational time. Hence, we identified limitations to address in future studies, notably the need to derive an explicit mathematical expression for the logit phase, which could: (i) mitigate numerical instability in more complex architectures (e.g., CNNs) while reducing computational overhead, and (ii) enable a systematic evaluation of CMG as an activation function across all layers. Furthermore, CMG-GLLF adopted as data transformation function enhances the accuracy of affinity-graph-based neuron segmentation. CMG-GLLF combines in a unique framework the ability of logistic and logit function to modulate signals or variables, covering a full spectrum of attenuation or amplification transformations. CMG-GLLF is flexible and trainable, has potential to advance machine learning models, and can inspire further applications in other data analysis challenges in different domains of science.

30 October 2025 in Preprints.org
Verified
Preprint
Open access
PDF file
WWWenjing Wu
Wenjing Wu
YZYingtao Zhang
Yingtao Zhang
JZJialin Zhao
Jialin Zhao
Carlo Vittorio CannistraciCarlo Vittorio Cannistraci
Carlo Vittorio Cannistraci
With the rapid development of large language models (LLMs), identifying efficient strategies for training such large-scale systems has become increasingly critical. Although LLMs have achieved remarkable success across diverse applications, the necessity of maintaining full dense matrices during pre-training has been questioned, giving rise to parameter-efficient sparse pre-training methods which retains parameter-efficiency in both training and inference. These methods can be further divided into connectivity sparse training and spectral sparse training, with dynamic connectivity sparse training and low-rank factorization emerging as representative approaches for the two branches. However, a unified framework that effectively combines the strengths of both has yet to be established. In this work, we observe that the cancellation effect between the sparse and low-rank branches may limit the expressivity of the model, manifesting as output conflicts when the two components are combined. To address this issue, we propose a novel scheme that integrates dynamic sparse training with low-rank training, introducing a simple yet effective alignment loss to mitigate the disagreement between the two branches and promote better collaboration. We validate this scheme by combining a representative dynamic sparse training method, CHTs, with low-rank training, resulting in a new parameter-efficient training approach termed CHTsL. The method is evaluated on LLaMA60M and LLaMA130M using the OpenWebText and C4 datasets, where only 10\%, 20\%, and 30\% of the parameters are preserved compared to dense training. Experimental results demonstrate that our proposed scheme effectively alleviates the cancellation effect and improves training stability and performance compared to the naive combination of sparse and low-rank components. Also, the new scheme enables CHTsL to consistently outperform other parameter-efficient sparse training methods under the same parameter budget, achieving performance most close to dense training.

4 January 2026 in Preprints.org
Verified
Preprint
Open access
PDF file
JZJialin Zhao
Jialin Zhao
AMAlessandro Muscoloni
Alessandro Muscoloni
UMUmberto Michieli
Umberto Michieli
YZYingtao Zhang
Yingtao Zhang
Carlo Vittorio CannistraciCarlo Vittorio Cannistraci
Carlo Vittorio Cannistraci
Many complex networks have partially observed or evolving connectivity, making link prediction a fundamental task. Topological link prediction infers missing links using only network topology, with applications in social, biological, and technological systems. The Cannistraci-Hebb (CH) theory provides a topological formulation of Hebbian learning, grounded on two pillars: (1) the minimization of external links within local communities, and (2) the path-based definition of local communities that capture homophilic (similarity-driven) interactions via paths of length 2 and synergetic (diversitydriven) interactions via paths of length 3. Building on this, we introduce the Cannistraci-Hebb Adaptive (CHA) network automata, an adaptive learning machine that automatically selects the optimal CH rule and path length to model each network. CHA unifies theoretical interpretability and data-driven adaptivity, bridging physics-inspired network science and machine intelligence. Across 1,269 networks from 14 domains, CHA consistently surpasses state-of-the-art methods—including SPM, SBM, graph embedding methods, and message-passing graph neural networks—while revealing the mechanistic principles governing link formation. Our code is available at https://github.com/biomedical-cybernetics/Cannistraci_Hebb_network_automata.