Illia Horenko

Illia Horenko

Auto-generated profile, based on publicly available information

Illia Horenko

Università della Svizzera Italiana, Faculty of Informatics, TI-6900 Lugano, Switzerland

Research Focus & Metrics

Academic Career Length

21Years (2006-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 August 2020 in Neural Computation
Verified
Journal Article
Closed access
IHIllia Horenko
Illia Horenko
Overfitting and treatment of small data are among the most challenging problems in machine learning (ML), when a relatively small data statistics size [ ] is not enough to provide a robust ML fit for a relatively large data feature dimension [ ]. Deploying a massively parallel ML analysis of generic classification problems for different [ ] and [ ], we demonstrate the existence of statistically significant linear overfitting barriers for common ML methods. The results reveal that for a robust classification of bioinformatics-motivated generic problems with the long short-term memory deep learning classifier (LSTM), one needs in the best case a statistics [ ] that is at least 13.8 times larger than the feature dimension [ ]. We show that this overfitting barrier can be breached at a 10[ ] fraction of the computational cost by means of the entropy-optimal scalable probabilistic approximations algorithm (eSPA), performing a joint solution of the entropy-optimal Bayesian network inference and feature space segmentation problems. Application of eSPA to experimental single cell RNA sequencing data exhibits a 30-fold classification performance boost when compared to standard bioinformatics tools and a 7-fold boost when compared to the deep learning LSTM classifier.

15 April 2022 in Neural Computation
Verified
Journal Article
Closed access
PDF file
EVEdoardo Vecchi
Edoardo Vecchi
LPLukáš Pospíšil
Lukáš Pospíšil
SASteffen Albrecht
Steffen Albrecht
Terence O'kaneTerence O'kane
Terence O'kane
IHIllia Horenko
Illia Horenko
Classification problems in the small data regime (with small data statistic T and relatively large feature space dimension D) impose challenges for the common machine learning (ML) and deep learning (DL) tools. The standard learning methods from these areas tend to show a lack of robustness when applied to data sets with significantly fewer data points than dimensions and quickly reach the overfitting bound, thus leading to poor performance beyond the training set. To tackle this issue, we propose eSPA+, a significant extension of the recently formulated entropy-optimal scalable probabilistic approximation algorithm (eSPA). Specifically, we propose to change the order of the optimization steps and replace the most computationally expensive subproblem of eSPA with its closed-form solution. We prove that with these two enhancements, eSPA+ moves from the polynomial to the linear class of complexity scaling algorithms. On several small data learning benchmarks, we show that the eSPA+ algorithm achieves a many-fold speed-up with respect to eSPA and even better performance results when compared to a wide array of ML and DL tools. In particular, we benchmark eSPA+ against the standard eSPA and the main classes of common learning algorithms in the small data regime: various forms of support vector machines, random forests, and long short-term memory algorithms. In all the considered applications, the common learning methods and eSPA are markedly outperformed by eSPA+, which achieves significantly higher prediction accuracy with an orders-of-magnitude lower computational cost.

31 January 2020 in Science Advances
Verified
Journal Article
Open access
PDF file
SGSusanne Gerber
Susanne Gerber
LPLukáš Pospíšil
Lukáš Pospíšil
MNM. Navandar
M. Navandar
IHI. Horenko
I. Horenko
Finding reliable discrete approximations of complex systems is a key prerequisite when applying many of the most popular modeling tools. Common discretization approaches (e.g., the very popular K-means clustering) are crucially limited in terms of quality, parallelizability, and cost. We introduce a low-cost improved quality scalable probabilistic approximation (SPA) algorithm, allowing for simultaneous data-driven optimal discretization, feature selection, and prediction. We prove its optimality, parallel efficiency, and a linear scalability of iteration cost. Cross-validated applications of SPA to a range of large realistic data classification and prediction problems reveal marked cost and performance improvements. For example, SPA allows the data-driven next-day predictions of resimulated surface temperatures for Europe with the mean prediction error of 0.75°C on a common PC (being around 40% better in terms of errors and five to six orders of magnitude cheaper than with common computational instruments used by the weather services).

29 December 2022 in Proceedings of the National Academy of Sciences
Verified
Journal Article
Open access
IHIllia Horenko
Illia Horenko
EVEdoardo Vecchi
Edoardo Vecchi
JKJuraj Kardoš
Juraj Kardoš
AWAndreas Wächter
Andreas Wächter
OSOlaf Schenk
Olaf Schenk
SGSusanne Gerber
Susanne Gerber
Regression learning is one of the long-standing problems in statistics, machine learning, and deep learning (DL). We show that writing this problem as a probabilistic expectation over (unknown) feature probabilities – thus increasing the number of unknown parameters and seemingly making the problem more complex—actually leads to its simplification, and allows incorporating the physical principle of entropy maximization. It helps decompose a very general setting of this learning problem (including discretization, feature selection, and learning multiple piece-wise linear regressions) into an iterative sequence of simple substeps, which are either analytically solvable or cheaply computable through an efficient second-order numerical solver with a sublinear cost scaling. This leads to the computationally cheap and robust non-DL second-order Sparse Probabilistic Approximation for Regression Task Analysis (SPARTAn) algorithm, that can be efficiently applied to problems with millions of feature dimensions on a commodity laptop, when the state-of-the-art learning tools would require supercomputers. SPARTAn is compared to a range of commonly used regression learning tools on synthetic problems and on the prediction of the El Niño Southern Oscillation, the dominant interannual mode of tropical climate variability. The obtained SPARTAn learners provide more predictive, sparse, and physically explainable data descriptions, clearly discerning the important role of ocean temperature variability at the thermocline in the equatorial Pacific. SPARTAn provides an easily interpretable description of the timescales by which these thermocline temperature features evolve and eventually express at the surface, thereby enabling enhanced predictability of the key drivers of the interannual climate.

1 July 2011 in Journal of the Atmospheric Sciences
Verified
Journal Article
Open access
IHIllia Horenko
Illia Horenko
Because of the mathematical and numerical limitations, standard statistical methods known from the literature are not applicable to inferring jump processes under exogenous influence. Such processes can be considered, for example, in the atmosphere (transitions between different cloud types) and in the ocean (phase transitions between water and ice). Reasons for these intrinsic limitations of standard methods are investigated and a method for the inference of discrete microscopic jump models based on macroscopic ensemble observations is presented. It significantly extends the recently developed methods of nonstationary Markov model parameterization (which are constrained to a single exogenous factor and to direct individual observations of the jump process realizations). The main advantage of the new method is the possibility of inference from indirect ensemble observations with multiple exogenous factors. Moreover, this method allows for a new possibility to test whether the available time series...

29 September 2014 in Proceedings of the National Academy of Sciences
Verified
Journal Article
Open access
PDF file
SGSusanne Gerber
Susanne Gerber
IHIllia Horenko
Illia Horenko
Discrete state models are a common tool of modeling in many areas. E.g., Markov state models as a particular representative of this model family became one of the major instruments for analysis and understanding of processes in molecular dynamics (MD). Here we extend the scope of discrete state models to the case of systematically missing scales, resulting in a nonstationary and nonhomogeneous formulation of the inference problem. We demonstrate how the recently developed tools of nonstationary data analysis and information theory can be used to identify the simultaneously most optimal (in terms of describing the given data) and most simple (in terms of complexity and causality) discrete state models. We apply the resulting formalism to a problem from molecular dynamics and show how the results can be used to understand the spatial and temporal causality information beyond the usual assumptions. We demonstrate that the most optimal explanation for the appropriately discretized/coarse-grained MD torsion angles data in a polypeptide is given by the causality that is localized both in time and in space, opening new possibilities for deploying percolation theory and stochastic subgridscale modeling approaches in the area of MD.