Mila > Publication > Recurrent Neural Networks > Regularizing RNNs by Stabilizing Activations

Regularizing RNNs

Recurrent Neural Networks
Nov 2015

Regularizing RNNs by Stabilizing Activations

Nov 2015

We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing the squared distance between successive hidden states’ norms. This penalty term is an effective regularizer for RNNs including LSTMs and IRNNs, improving performance on character-level language modelling and phoneme recognition, and outperforming weight noise and dropout. We set state of the art (17.5% PER) for an RNN on the TIMIT phoneme recognition task, without using beam-search. With this penalty term, IRNN can achieve similar performance to LSTM on language modelling, although adding the penalty term to the LSTM results in superior performance. Our penalty term also prevents the exponential growth of IRNN’s activations outside of their training horizon, allowing them to generalize to much longer sequences.


David Krueger, Roland Memisevic, Regularizing RNNs by Stabilizing Activations, in: International Conference on Learning Representations, 2016


[arXiv:1511.08400] [ICLR 2016]

Linked Profiles