Portrait of Stefan Horoi is unavailable

Stefan Horoi

stefan.horoi@mila.quebec
PhD - Université de Montréal
Supervisor
Guy Wolf
Research Topics
Deep Learning
Representation Learning
Website
Google Scholar
GitHub

Publications

Exploring the Geometry and Topology of Neural Network Loss Landscapes
Stefan Horoi
Jessie Huang
Bastian Rieck
Guillaume Lajoie
Guy Wolf
Smita Krishnaswamy
Recent work has established clear links between the generalization performance of trained neural networks and the geometry of their loss lan… (see more)dscape near the local minima to which they converge. This suggests that qualitative and quantitative examination of the loss landscape geometry could yield insights about neural network generalization performance during training. To this end, researchers have proposed visualizing the loss landscape through the use of simple dimensionality reduction techniques. However, such visualization methods have been limited by their linear nature and only capture features in one or two dimensions, thus restricting sampling of the loss landscape to lines or planes. Here, we expand and improve upon these in three ways. First, we present a novel "jump and retrain" procedure for sampling relevant portions of the loss landscape. We show that the resulting sampled data holds more meaningful information about the network's ability to generalize. Next, we show that non-linear dimensionality reduction of the jump and retrain trajectories via PHATE, a trajectory and manifold-preserving method, allows us to visualize differences between networks that are generalizing well vs poorly. Finally, we combine PHATE trajectories with a computational homology characterization to quantify trajectory differences.
2021-01-30
arXiv (preprint)
doi.org
arxiv.org
Advantages of biologically-inspired adaptive neural activation in RNNs during learning
Victor Geadah
Giancarlo Kerg
Stefan Horoi
Guy Wolf
Guillaume Lajoie
Dynamic adaptation in single-neuron response plays a fundamental role in neural coding in biological neural networks. Yet, most neural activ… (see more)ation functions used in artificial networks are fixed and mostly considered as an inconsequential architecture choice. In this paper, we investigate nonlinear activation function adaptation over the large time scale of learning, and outline its impact on sequential processing in recurrent neural networks. We introduce a novel parametric family of nonlinear activation functions, inspired by input-frequency response curves of biological neurons, which allows interpolation between well-known activation functions such as ReLU and sigmoid. Using simple numerical experiments and tools from dynamical systems and information theory, we study the role of neural activation features in learning dynamics. We find that activation adaptation provides distinct task-specific solutions and in some cases, improves both learning speed and performance. Importantly, we find that optimal activation features emerging from our parametric family are considerably different from typical functions used in the literature, suggesting that exploiting the gap between these usual configurations can help learning. Finally, we outline situations where neural activation adaptation alone may help mitigate changes in input statistics in a given task, suggesting mechanisms for transfer learning optimization.
2020-06-21
ArXiv (preprint)
doi.org
arxiv.org