Portrait de Alessandro Sordoni

Alessandro Sordoni

Membre industriel principal
Professeur associé, Université de Montréal, Département d'informatique et de recherche opérationnelle
Chercheur scientifique, Microsoft Research Montréal
Sujets de recherche
Grands modèles de langage (LLM)
Raisonnement
Traitement du langage naturel

Biographie

Je suis chercheur principal à Microsoft Research Montréal. J'ai obtenu un doctorat de l'Université de Montréal sous la direction de Jian-Yun Nie, en étudiant comment représenter efficacement les documents et les requêtes pour la recherche d'information. Présentement, je m’intéresse à l'étude de l'efficacité de l'apprentissage et de la généralisation systématique dans les grands modèles actuels d'apprentissage profond. Mes intérêts s'étendent à l'apprentissage non supervisé et à l'apprentissage à petite échelle, en particulier dans le domaine du langage naturel.

Étudiants actuels

Zhan Su
Collaborateur·rice alumni - University of Copenhagen

Publications

Learning Hierarchical Structures On-The-Fly with a Recurrent-Recursive Model for Sequences
Athul Paul Jacob
Zhouhan Lin
Alessandro Sordoni
Yoshua Bengio
We propose a hierarchical model for sequential data that learns a tree on-the-fly, i.e. while reading the sequence. In the model, a recurren… (voir plus)t network adapts its structure and reuses recurrent weights in a recursive manner. This creates adaptive skip-connections that ease the learning of long-term dependencies. The tree structure can either be inferred without supervision through reinforcement learning, or learned in a supervised manner. We provide preliminary experiments in a novel Math Expression Evaluation (MEE) task, which is created to have a hierarchical tree structure that can be used to study the effectiveness of our model. Additionally, we test our model in a well-known propositional logic and language modelling tasks. Experimental results have shown the potential of our approach.
2018-06-30
Rep4NLP@ACL (publié)
doi.org
Straight to the Tree: Constituency Parsing with Neural Syntactic Distance
Yikang Shen
Zhouhan Lin
Athul Paul Jacob
Alessandro Sordoni
Aaron Courville
Yoshua Bengio
In this work, we propose a novel constituency parsing scheme. The model predicts a vector of real-valued scalars, named syntactic distances,… (voir plus) for each split position in the input sentence. The syntactic distances specify the order in which the split points will be selected, recursively partitioning the input, in a top-down fashion. Compared to traditional shift-reduce parsing schemes, our approach is free from the potential problem of compounding errors, while being faster and easier to parallelize. Our model achieves competitive performance amongst single model, discriminative parsers in the PTB dataset and outperforms previous models in the CTB dataset.
2018-06-30
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (publié)
doi.org
arxiv.org
Towards Text Generation with Adversarially Learned Neural Outlines.
Sandeep Subramanian
Sai Rajeswar
Alessandro Sordoni
Adam Trischler
Aaron C. Courville
Chris Pal
2017-12-31
Advances in Neural Information Processing Systems 31 (NeurIPS 2018) (publié)
dblp.uni-trier.de
Twin Networks: Matching the Future for Sequence Generation
Dmitriy Serdyuk
Nan Rosemary Ke
Alessandro Sordoni
Adam Trischler
Christopher Pal
Yoshua Bengio
We propose a simple technique for encouraging generative RNNs to plan ahead. We train a "backward" recurrent network to generate a given seq… (voir plus)uence in reverse order, and we encourage states of the forward model to predict cotemporal states of the backward model. The backward network is used only during training, and plays no role during sampling or inference. We hypothesize that our approach eases modeling of long-term dependencies by implicitly forcing the forward states to hold information about the longer-term future (as contained in the backward states). We show empirically that our approach achieves 9% relative improvement for a speech recognition task, and achieves significant improvement on a COCO caption generation task.
2017-12-31
ICLR (Poster) (publié)
openreview.net
openreview.net
Twin Networks: Using the Future as a Regularizer
Dmitriy Serdyuk
Nan Rosemary Ke
Alessandro Sordoni
Christopher Pal
Yoshua Bengio
Being able to model long-term dependencies in sequential data, such as text, has been among the long-standing challenges of recurrent neural… (voir plus) networks (RNNs). This issue is strictly related to the absence of explicit planning in current RNN architectures. More explicitly, the RNNs are trained to predict only the next token given previous ones. In this paper, we introduce a simple way of encouraging the RNNs to plan for the future. In order to accomplish this, we introduce an additional neural network which is trained to generate the sequence in reverse order, and we require closeness between the states of the forward RNN and backward RNN that predict the same token. At each step, the states of the forward RNN are required to match the future information contained in the backward states. We hypothesize that the approach eases modeling of long-term dependencies thus helping in generating more globally consistent samples. The model trained with conditional generation for a speech recognition task achieved 12\% relative improvement (CER of 6.7 compared to a baseline of 7.6).
2017-08-21
ArXiv (prépublication)
arxiv.org
arxiv.org
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
Iulian V. Serban
Alessandro Sordoni
Ryan Lowe
Laurent Charlin
Joelle Pineau
Aaron Courville
Yoshua Bengio
Sequential data often possesses a hierarchical structure with complex dependencies between subsequences, such as found between the utterance… (voir plus)s in a dialogue. In an effort to model this kind of generative process, we propose a neural network-based generative architecture, with latent stochastic variables that span a variable number of time steps. We apply the proposed model to the task of dialogue response generation and compare it with recent neural network architectures. We evaluate the model performance through automatic evaluation metrics and by carrying out a human evaluation. The experiments demonstrate that our model improves upon recently proposed models and that the latent variables facilitate the generation of long outputs and maintain the context.
2017-02-11
Proceedings of the AAAI Conference on Artificial Intelligence (publié)
doi.org
arxiv.org
Z-Forcing: Training Stochastic Recurrent Networks
Anirudh Goyal
Alessandro Sordoni
Marc-Alexandre Côté
Nan Rosemary Ke
Yoshua Bengio
Many efforts have been devoted to training generative latent variable models with autoregressive decoders, such as recurrent neural networks… (voir plus) (RNN). Stochastic recurrent models have been successful in capturing the variability observed in natural sequential data such as speech. We unify successful ideas from recently proposed architectures into a stochastic recurrent model: each step in the sequence is associated with a latent variable that is used to condition the recurrent dynamics for future steps. Training is performed with amortized variational inference where the approximate posterior is augmented with a RNN that runs backward through the sequence. In addition to maximizing the variational lower bound, we ease training of the latent variables by adding an auxiliary cost which forces them to reconstruct the state of the backward recurrent network. This provides the latent variables with a task-independent objective that enhances the performance of the overall model. We found this strategy to perform better than alternative approaches such as KL annealing. Although being conceptually simple, our model achieves state-of-the-art results on standard speech benchmarks such as TIMIT and Blizzard and competitive performance on sequential MNIST. Finally, we apply our model to language modeling on the IMDB dataset where the auxiliary cost helps in learning interpretable latent variables. Source Code: https://github.com/anirudh9119/zforcing_nips17
2016-12-31
Advances in Neural Information Processing Systems 30 (NIPS 2017) (publié)
doi.org
arxiv.org
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Iulian V. Serban
Alessandro Sordoni
Yoshua Bengio
Aaron Courville
Joelle Pineau
We investigate the task of building open domain, conversational dialogue systems based on large dialogue corpora using generative models. Ge… (voir plus)nerative models produce system responses that are autonomously generated word-by-word, opening up the possibility for realistic, flexible interactions. In support of this goal, we extend the recently proposed hierarchical recurrent encoder-decoder neural network to the dialogue domain, and demonstrate that this model is competitive with state-of-the-art neural language models and back-off n-gram models. We investigate the limitations of this and similar approaches, and show how its performance can be improved by bootstrapping the learning from a larger question-answer pair corpus and from pretrained word embeddings.
2016-03-04
Proceedings of the AAAI Conference on Artificial Intelligence (publié)
doi.org
arxiv.org