Sourya Dey

Dr. Sourya Dey

orcid

Research Focus & Metrics

Academic Career Length

10Years (2017-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
12 April 2019 in IEEE Journal on Emerging and Selected Topics in Circuits and Systems
Verified
Journal Article
Closed access
SDSourya Dey
Sourya Dey
KHKuan-Wen Huang
Kuan-Wen Huang
PBPeter Anthony Beerel
Peter Anthony Beerel
KCKeith M. Chugg
Keith M. Chugg
Neural networks have proven to be extremely powerful tools for modern artificial intelligence applications, but computational and storage complexity remain limiting factors. This paper presents two compatible contributions towards reducing the time, energy, computational, and storage complexities associated with multilayer perceptrons. Pre-defined sparsity is proposed to reduce the complexity during both training and inference, regardless of the implementation platform. Our results show that storage and computational complexity can be reduced by factors greater than 5X without significant performance loss. The second contribution is an architecture for hardware acceleration that is compatible with pre-defined sparsity. This architecture supports both training and inference modes and is flexible in the sense that it is not tied to a specific number of neurons. For example, this flexibility implies that various sized neural networks can be supported on various sized Field Programmable Gate Array (FPGA)s.

24 October 2017
Verified
Book Chapter
Closed access
PDF file
SDSourya Dey
Sourya Dey
YSYinan Shao
Yinan Shao
KCKeith M. Chugg
Keith M. Chugg
PBPeter A. Beerel
Peter A. Beerel

8 May 2021 in SN Computer Science
Verified
Journal Article
Closed access
PDF file
SDSourya Dey
Sourya Dey
SBSara Babakniya
Sara Babakniya
SKSaikrishna C. Kanala
Saikrishna C. Kanala
MPMarco Paolieri
Marco Paolieri
LGLeana Golubchik
Leana Golubchik
KCKeith M. Chugg
Keith M. Chugg

17 October 2025 in Mathematics
Verified
Journal Article
Open access
PDF file
MRMichael Robinson
Michael Robinson
SDSourya Dey
Sourya Dey
TKTaisa Kushner
Taisa Kushner
Some large language models (LLMs) are open source and are therefore fully open for scientific study. However, many LLMs are proprietary, and their internals are hidden, which hinders the ability of the research community to study their behavior under controlled conditions. For instance, the token input embedding specifies an internal vector representation of each token used by the model. If the token input embedding is hidden, latent semantic information about the set of tokens is unavailable to researchers. This article presents a general and flexible method for prompting an LLM to reveal its token input embedding, even if this information is not published with the model. Moreover, this article provides strong theoretical justification—a mathematical proof for generic LLMs—for why this method should be expected to work. If the LLM can be prompted systematically and certain benign conditions about the quantity of data collected from the responses are met, the topology of the token embedding is recovered. With this method in hand, we demonstrate its effectiveness by recovering the token subspace of the Llemma-7BLLM. We demonstrate the flexibility of this method by performing the recovery at three different times, each using the same algorithm applied to different information collected from the responses. While the prompting can be a performance bottleneck depending on the size and complexity of the LLM, the recovery runs within a few hours on a typical workstation. The results of this paper apply not only to LLMs but also to general nonlinear autoregressive processes.

23 June 2023 in arXiv
Verified
Preprint
Open access
PDF file
SDSourya Dey
Sourya Dey
EDEric Davis
Eric Davis
We present DLKoopman -- a software package for Koopman theory that uses deep learning to learn an encoding of a nonlinear dynamical system into a linear space, while simultaneously learning the linear dynamics. While several previous efforts have either restricted the ability to learn encodings, or been bespoke efforts designed for specific systems, DLKoopman is a generalized tool that can be applied to data-driven learning and optimization of any dynamical system. It can either be trained on data from individual states (snapshots) of a system and used to predict its unknown states, or trained on data from trajectories of a system and used to predict unknown trajectories for new initial states. DLKoopman is available on the Python Package Index (PyPI) as 'dlkoopman', and includes extensive documentation and tutorials. Additional contributions of the package include a novel metric called Average Normalized Absolute Error for evaluating performance, and a ready-to-use hyperparameter search module for improving performance.

11 October 2024 in arXiv
Verified
Preprint
Open access
PDF file
MRMichael Robinson
Michael Robinson
SDSourya Dey
Sourya Dey
SSShauna Sweet
Shauna Sweet
Large language models encode the correlational structure present in natural language by fitting segments of utterances (tokens) into a high dimensional ambient latent space upon which the models then operate. We assert that in order to develop a foundational, first-principles understanding of the behavior and limitations of large language models, it is crucial to understand the topological and geometric structure of this token subspace. In this article, we present estimators for the dimension and Ricci scalar curvature of the token subspace, and apply it to three open source large language models of moderate size: GPT2, LLEMMA7B, and MISTRAL7B. In all three models, using these measurements, we find that the token subspace is not a manifold, but is instead a stratified manifold, where on each of the individual strata, the Ricci curvature is significantly negative. We additionally find that the dimension and curvature correlate with generative fluency of the models, which suggest that these findings have implications for model behavior.