Jialin Mao

Jialin Mao

Auto-generated profile, based on publicly available information

Jialin Mao

University of Pennsylvania

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
12 March 2024 in Proceedings of the National Academy of Sciences
Verified
Journal Article
Open access
JMJialin Mao
Jialin Mao
IGItay Griniasty
Itay Griniasty
HTHan Kheng Teoh
Han Kheng Teoh
RRRahul Ramesh
Rahul Ramesh
RYRubing Yang
Rubing Yang
PCPratik Chaudhari
Pratik Chaudhari
We develop information-geometric techniques to analyze the trajectories of the predictions of deep networks during training. By examining the underlying high-dimensional probabilistic models, we reveal that the training process explores an effectively low-dimensional manifold. Networks with a wide range of architectures, sizes, trained using different optimization methods, regularization techniques, data augmentation techniques, and weight initializations lie on the same manifold in the prediction space. We study the details of this manifold to find that networks with different architectures follow distinguishable trajectories, but other factors have a minimal influence; larger networks train along a similar manifold as that of smaller networks, just faster; and networks initialized at very different parts of the prediction space converge to the solution along a similar manifold.

21 January 2026 in Physical Review E
Verified
Journal Article
Closed access
JMJialin Mao
Jialin Mao
IGItay Griniasty
Itay Griniasty
YSYan Sun
Yan Sun
MTMark K Transtrum
Mark K Transtrum
JSJames P. Sethna
James P. Sethna
PCPratik Chaudhari
Pratik Chaudhari
Recent experiments have shown that training trajectories of multiple deep neural networks with different architectures, optimization algorithms, hyperparameter settings, and regularization methods evolve on a remarkably low-dimensional "hyperribbon-like" manifold in the space of probability distributions. Inspired by the similarities in the training trajectories of deep networks and linear networks, we analytically characterize this phenomenon for the latter. We show, using tools in dynamical systems theory, that the geometry of this low-dimensional manifold is controlled by (i) the decay rate of the eigenvalues of the input correlation matrix of the training data, (ii) the relative scale of the ground-truth output to the weights at the beginning of training, and (iii) the number of steps of gradient descent. By analytically computing and bounding the contributions of these quantities, we characterize phase boundaries of the region where hyperribbons are to be expected. We also extend our analysis to kernel machines and linear models that are trained with stochastic gradient descent.