9 October 2023 in Neural Networks
The problem of reducing processing time of large deep learning models is a fundamental challenge in many real-world applications. Early exit methods strive towards this goal by attaching additional Internal Classifiers (ICs) to intermediate layers of a neural network. ICs can quickly return predictions for easy examples and, as a result, reduce the average inference time of the whole model. However, if a particular IC does not decide to return an answer early, its predictions are discarded, with its computations effectively being wasted. To solve this issue, we introduce Zero Time Waste (ZTW), a novel approach in which each IC reuses predictions returned by its predecessors by (1) adding direct connections between ICs and (2) combining previous outputs in an ensemble-like manner. We conduct extensive experiments across various multiple modes, datasets, and architectures to demonstrate that ZTW achieves a significantly better accuracy vs. inference time trade-off than other early exit methods. On the ImageNet dataset, it obtains superior results over the best baseline method in 11 out of 16 cases, reaching up to 5 percentage points of improvement on low computational budgets.
10 March 2023 in IEEE Transactions on Neural Networks and Learning Systems
Interpolating between points is a problem connected simultaneously with finding geodesics and study of generative models. In the case of geodesics, we search for the curves with the shortest length, while in the case of generative models, we typically apply linear interpolation in the latent space. However, this interpolation uses implicitly the fact that Gaussian is unimodal. Thus, the problem of interpolating in the case when the latent density is non-Gaussian is an open problem. In this article, we present a general and unified approach to interpolation, which simultaneously allows us to search for geodesics and interpolating curves in latent space in the case of arbitrary density. Our results have a strong theoretical background based on the introduced quality measure of an interpolating curve. In particular, we show that maximizing the quality measure of the curve can be equivalently understood as a search of geodesic for a certain redefinition of the Riemannian metric on the space. We provide examples in three important cases. First, we show that our approach can be easily applied to finding geodesics on manifolds. Next, we focus our attention in finding interpolations in pretrained generative models. We show that our model effectively works in the case of arbitrary density. Moreover, we can interpolate in the subset of the space consisting of data possessing a given feature. The last case is focused on finding interpolation in the space of chemical compounds.
28 September 2020 in PsyArXiv
Background: Some studies suggest that as much as 40% of all causes of death in a group of patients with schizophrenia can be attributed to suicides and compared with the general population, patients with schizophrenia have an 8.5-fold greater suicide risk (SR). There is a vital need for accurate and reliable methods to predict the SR among patients with schizophrenia based on biological measures. However, it is unknown whether the suicidal risk in schizophrenia can be related to alterations in spontaneous brain activity, or if the resting-state functional magnetic resonance imaging (rsfMRI) measures can be used alongside machine learning (ML) algorithms in order to identify patients with SR.Methods: Fifty-nine participants including patients with schizophrenia with and without SR as well as age and gender-matched healthy underwent 13 minutes of resting-state functional magnetic resonance imaging. Both static and dynamic indexes of the amplitude of low-frequency fluctuation (ALFF), the fractional amplitude of low-frequency fluctuations (fALFF), regional homogeneity as well as functional connectivity (FC) was calculated and used as an input for five machine learning algorithms: Gradient boosting (GB), LASSO, Logistic Regression (LR), Random Forest and Support Vector Machine. Results: All groups revealed different internetwork functional connectivity in ventral DMN and anterior SN. The best performance was reached for the LASSO applied to FC with an accuracy of 70% and AUROC of 0.76 (p<0.05). Significant classification ability was also reached for GB and LR using fALFF and ALFF measures.Conclusion: Our findings suggest that SR in schizophrenia can be seen on the level of DMN and SN functional connectivity alterations. ML algorithms were able to significantly differentiate SR patients. Our results could be useful in developing neuromarkers of SR in schizophrenia based on non-invasive rsfMRI.