Barry Koren

Barry Koren

Auto-generated profile, based on publicly available information

Barry Koren

Eindhoven University of Technology

Research Focus & Metrics

Academic Career Length

13Years (2014-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 December 2020 in Computer Methods in Applied Mechanics and Engineering
Verified
Journal Article
Closed access
SLStephan Benjamin Lunowa
Stephan Benjamin Lunowa
IPIuliu Sorin Pop
Iuliu Sorin Pop
BKBarry Koren
Barry Koren



20 December 2023 in Journal of Optimization Theory and Applications
Verified
Journal Article
Closed access
PDF file
LXLu Xia
Lu Xia
SMStefano Massei
Stefano Massei
MHMichiel E. Hochstenbach
Michiel E. Hochstenbach
BKBarry Koren
Barry Koren

31 May 2020 in arXiv
Verified
Preprint
Open access
PDF file
LXLu Xia
Lu Xia
MAMartijn Anthonissen
Martijn Anthonissen
MHMichiel Hochstenbach
Michiel Hochstenbach
BKBarry Koren
Barry Koren
Due to the limited number of bits in floating-point or fixed-point arithmetic, rounding is a necessary step in many computations. Although rounding methods can be tailored for different applications, round-off errors are generally unavoidable. When a sequence of computations is implemented, round-off errors may be magnified or accumulated. The magnification of round-off errors may cause serious failures. Stochastic rounding (SR) was introduced as an unbiased rounding method, which is widely employed in, for instance, the training of neural networks (NNs), showing a promising training result even in low-precision computations. Although the employment of SR in training NNs is consistently increasing, the error analysis of SR is still to be improved. Additionally, the unbiased rounding results of SR are always accompanied by large variances. In this study, some general properties of SR are stated and proven. Furthermore, an upper bound of rounding variance is introduced and validated. Two new probability distributions of SR are proposed to study the trade-off between variance and bias, by solving a multiple objective optimization problem. In the simulation study, the rounding variance, bias, and relative errors of SR are studied for different operations, such as summation, square root calculation through Newton iteration and inner product computation, with specific rounding precision.

25 February 2023 in arXiv
Verified
Preprint
Open access
PDF file
LXLu Xia
Lu Xia
SMStefano Massei
Stefano Massei
MHMichiel E. Hochstenbach
Michiel E. Hochstenbach
BKBarry Koren
Barry Koren
When implementing the gradient descent method in low precision, the employment of stochastic rounding schemes helps to prevent stagnation of convergence caused by the vanishing gradient effect. Unbiased stochastic rounding yields zero bias by preserving small updates with probabilities proportional to their relative magnitudes. This study provides a theoretical explanation for the stagnation of the gradient descent method in low-precision computation. Additionally, we propose two new stochastic rounding schemes that trade the zero bias property with a larger probability to preserve small gradients. Our methods yield a constant rounding bias that, on average, lies in a descent direction. For convex problems, we prove that the proposed rounding methods typically have a beneficial effect on the convergence rate of gradient descent. We validate our theoretical analysis by comparing the performances of various rounding schemes when optimizing a multinomial logistic regression model and when training a simple neural network with an 8-bit floating-point format.