1 August 2005 in Neurocomputing
26 July 2021 in Neural Computation
The Fisher information matrix (FIM) plays an essential role in statistics and machine learning as a Riemannian metric tensor or a component of the Hessian matrix of loss functions. Focusing on the FIM and its variants in deep neural networks (DNNs), we reveal their characteristic scale dependence on the network width, depth, and sample size when the network has random weights and is sufficiently wide. This study covers two widely used FIMs for regression with linear output and for classification with softmax output. Both FIMs asymptotically show pathological eigenvalue spectra in the sense that a small number of eigenvalues become large outliers depending on the width or sample size, while the others are much smaller. It implies that the local shape of the parameter space or loss landscape is very sharp in a few specific directions while almost flat in the other directions. In particular, the softmax output disperses the outliers and makes a tail of the eigenvalue density spread from the bulk. We also show that pathological spectra appear in other variants of FIMs: one is the neural tangent kernel; another is a metric for the input signal and feature space that arises from feedforward signal propagation. Thus, we provide a unified perspective on the FIM and its variants that will lead to more quantitative understanding of learning in large-scale DNNs.
27 June 2019 in Science and Technology of Advanced Materials
We introduce a spectrum-adapted expectation-maximization (EM) algorithm for high-throughput analysis of a large number of spectral datasets by considering the weight of the intensity corresponding to the measurement energy steps. Proposed method was applied to synthetic data in order to evaluate the performance of the analysis accuracy and calculation time. Moreover, the proposed method was performed to the spectral data collected from graphene and MoS2 field-effect transistors devices. The calculation completed in less than 13.4 s per set and successfully detected systematic peak shifts of the C 1s in graphene and S 2p in MoS2 peaks. This result suggests that the proposed method can support the investigation of peak shift with two advantages: (1) a large amount of data can be processed at high speed; and (2) stable and automatic calculation can be easily performed. Graphical Abstract
1 January 2006 in Lecture Notes in Computer Science