Portrait de Tim Cooijmans n'est pas disponible

Tim Cooijmans

Alumni

Publications

Advantage Alignment Algorithms
Juan Duque
Milad Aghajohari
Tim Cooijmans
Razvan Ciuca
Tianyu Zhang
Gauthier Gidel
Aaron Courville
The growing presence of artificially intelligent agents in everyday decision-making, from LLM assistants to autonomous vehicles, hints at a … (voir plus)future in which conflicts may arise from each agent optimizing individual interests. In general-sum games these conflicts are apparent, where naive Reinforcement Learning agents get stuck in Pareto-suboptimal Nash equilibria. Consequently, opponent shaping has been introduced as a method with success at finding socially beneficial equilibria in social dilemmas. In this work, we introduce Advantage Alignment, a family of algorithms derived from first principles that perform opponent shaping efficiently and intuitively. This is achieved by aligning the advantages of conflicting agents in a given game by increasing the probability of mutually-benefiting actions. We prove that existing opponent shaping methods, including LOLA and LOQA, implicitly perform Advantage Alignment. Compared to these works, Advantage Alignment mathematically simplifies the formulation of opponent shaping and seamlessly works for continuous action domains. We also demonstrate the effectiveness of our algorithm in a wide range of social dilemmas, achieving state of the art results in each case, including a social dilemma version of the Negotiation Game.
2025-01-21
International Conference on Learning Representations (présentation orale)
doi.org
openreview.net
Adaptive Accompaniment with ReaLchords
Yusong Wu
Tim Cooijmans
Kyle Kastner
Adam Roberts
Ian Simon
Alexander Scarlatos
Chris Donahue
Cassie Tarakajian
Shayegan Omidshafiei
Aaron Courville
Pablo Samuel Castro
Natasha Jaques
Cheng-Zhi Anna Huang
Jamming requires coordination, anticipation, and collaborative creativity between musicians. Current generative models of music produce expr… (voir plus)essive output but are not able to generate in an \emph{online} manner, meaning simultaneously with other musicians (human or otherwise). We propose ReaLchords, an online generative model for improvising chord accompaniment to user melody. We start with an online model pretrained by maximum likelihood, and use reinforcement learning to finetune the model for online use. The finetuning objective leverages both a novel reward model that provides feedback on both harmonic and temporal coherency between melody and chord, and a divergence term that implements a novel type of distillation from a teacher model that can see the future melody. Through quantitative experiments and listening tests, we demonstrate that the resulting model adapts well to unfamiliar input and produce fitting accompaniment. ReaLchords opens the door to live jamming, as well as simultaneous co-creation in other modalities.
2024-07-07
Proceedings of the 41st International Conference on Machine Learning (publié)
doi.org
proceedings.mlr.press
Best Response Shaping
Milad Aghajohari
Tim Cooijmans
Juan Agustin Duque
Shunichi Akatsuka
Aaron Courville
We investigate the challenge of multi-agent deep reinforcement learning in partially competitive environments, where traditional methods str… (voir plus)uggle to foster reciprocity-based cooperation. LOLA and POLA agents learn reciprocity-based cooperative policies by differentiation through a few look-ahead optimization steps of their opponent. However, there is a key limitation in these techniques. Because they consider a few optimization steps, a learning opponent that takes many steps to optimize its return may exploit them. In response, we introduce a novel approach, Best Response Shaping (BRS), which differentiates through an opponent approximating the best response, termed the "detective." To condition the detective on the agent's policy for complex games we propose a state-aware differentiable conditioning mechanism, facilitated by a question answering (QA) method that extracts a representation of the agent based on its behaviour on specific environment states. To empirically validate our method, we showcase its enhanced performance against a Monte Carlo Tree Search (MCTS) opponent, which serves as an approximation to the best response in the Coin Game. This work expands the applicability of multi-agent RL in partially competitive environments and provides a new pathway towards achieving improved social welfare in general sum games.
2024-05-13
rl-conference.cc/RLC/2024/Conference (publié)
openreview.net
openreview.net
LOQA: Learning with Opponent Q-Learning Awareness
Milad Aghajohari
Juan Agustin Duque
Tim Cooijmans
Aaron Courville
In various real-world scenarios, interactions among agents often resemble the dynamics of general-sum games, where each agent strives to opt… (voir plus)imize its own utility. Despite the ubiquitous relevance of such settings, decentralized machine learning algorithms have struggled to find equilibria that maximize individual utility while preserving social welfare. In this paper we introduce Learning with Opponent Q-Learning Awareness (LOQA), a novel, decentralized reinforcement learning algorithm tailored to optimizing an agent's individual utility while fostering cooperation among adversaries in partially competitive environments. LOQA assumes the opponent samples actions proportionally to their action-value function Q. Experimental results demonstrate the effectiveness of LOQA at achieving state-of-the-art performance in benchmark scenarios such as the Iterated Prisoner's Dilemma and the Coin Game. LOQA achieves these outcomes with a significantly reduced computational footprint, making it a promising approach for practical multi-agent applications.
2024-01-15
ICLR.cc/2024/Conference (poster)
doi.org
openreview.net
Meta-Value Learning: a General Framework for Learning with Learning Awareness
Tim Cooijmans
Milad Aghajohari
Aaron Courville
2023-07-16
ArXiv (prépublication)
doi.org
openreview.net
Learning with Learning Awareness using Meta-Values
Tim Cooijmans
Milad Aghajohari
Aaron Courville
2023-06-18
ICML.cc/2023/Workshop/Frontiers4LCD (publié)
openreview.net
openreview.net
SUNMASK: Mask Enhanced Control in Step Unrolled Denoising Autoencoders
Kyle Kastner
Tim Cooijmans
Yusong Wu
Aaron Courville
2022-12-31
EvoMUSART@EvoStar (publié)
doi.org
openreview.net
MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling
Yusong Wu
Ethan Manilow
Yi Deng
Rigel Swavely
Kyle Kastner
Tim Cooijmans
Aaron Courville
Cheng-Zhi Anna Huang
Jesse Engel
Musical expression requires control of both what notes are played, and how they are performed. Conventional audio synthesizers provide detai… (voir plus)led expressive controls, but at the cost of realism. Black-box neural audio synthesis and concatenative samplers can produce realistic audio, but have few mechanisms for control. In this work, we introduce MIDI-DDSP a hierarchical model of musical instruments that enables both realistic neural audio synthesis and detailed user control. Starting from interpretable Differentiable Digital Signal Processing (DDSP) synthesis parameters, we infer musical notes and high-level properties of their expressive performance (such as timbre, vibrato, dynamics, and articulation). This creates a 3-level hierarchy (notes, performance, synthesis) that affords individuals the option to intervene at each level, or utilize trained priors (performance given notes, synthesis given performance) for creative assistance. Through quantitative experiments and listening tests, we demonstrate that this hierarchy can reconstruct high-fidelity audio, accurately predict performance attributes for a note sequence, independently manipulate the attributes of a given performance, and as a complete system, generate realistic audio from a novel note sequence. By utilizing an interpretable hierarchy, with multiple levels of granularity, MIDI-DDSP opens the door to assistive tools to empower individuals across a diverse range of musical experience.
2022-01-27
ICLR.cc/2022/Conference (présentation orale)
doi.org
openreview.net
Counterpoint by Convolution
Cheng-Zhi Anna Huang
Tim Cooijmans
Adam Roberts
Aaron Courville
Douglas Eck
Machine learning models of music typically break down the task of composition into a chronological process, composing a piece of music in a … (voir plus)single pass from beginning to end. On the contrary, human composers write music in a nonlinear fashion, scribbling motifs here and there, often revisiting choices previously made. We explore the use of blocked Gibbs sampling as an analogue to the human approach, and introduce Coconet, a convolutional neural network in the NADE family of generative models. Despite ostensibly sampling from the same distribution as the NADE ancestral sampling procedure, we find that a blocked Gibbs approach significantly improves sample quality. We provide evidence that this is due to some conditional distributions being poorly modeled. Moreover, we show that even the cheap approximate blocked Gibbs procedure from Yao et al. (2014) yields better samples than ancestral sampling. We demonstrate the versatility of our method on unconditioned polyphonic music generation.
2019-03-17
ArXiv (prépublication)
openreview.net
openreview.net
Harmonic Recomposition using Conditional Autoregressive Modeling
Kyle Kastner
Rithesh Kumar
Tim Cooijmans
Aaron Courville
We demonstrate a conditional autoregressive pipeline for efficient music recomposition, based on methods presented in van den Oord et al.(20… (voir plus)17). Recomposition (Casal & Casey, 2010) focuses on reworking existing musical pieces, adhering to structure at a high level while also re-imagining other aspects of the work. This can involve reuse of pre-existing themes or parts of the original piece, while also requiring the flexibility to generate new content at different levels of granularity. Applying the aforementioned modeling pipeline to recomposition, we show diverse and structured generation conditioned on chord sequence annotations.
2018-11-17
ArXiv (prépublication)
arxiv.org
arxiv.org
Recurrent Batch Normalization
Tim Cooijmans
Nicolas Ballas
César Laurent
Caglar Gulçehre
Aaron Courville
We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks. Whereas previous works… (voir plus) only apply batch normalization to the input-to-hidden transformation of RNNs, we demonstrate that it is both possible and beneficial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps. We evaluate our proposal on various sequential problems such as sequence classification, language modeling and question answering. Our empirical results show that our batch-normalized LSTM consistently leads to faster convergence and improved generalization.
2016-12-31
ICLR.cc/2017/conference (poster)
doi.org
openreview.net
Dynamic Capacity Networks
Amjad Almahairi
Nicolas Ballas
Tim Cooijmans
Yin Zheng
Hugo Larochelle
Aaron Courville
We introduce the Dynamic Capacity Network (DCN), a neural network that can adaptively assign its capacity across different portions of the i… (voir plus)nput data. This is achieved by combining modules of two types: low-capacity sub-networks and high-capacity sub-networks. The low-capacity sub-networks are applied across most of the input, but also provide a guide to select a few portions of the input on which to apply the high-capacity sub-networks. The selection is made using a novel gradient-based attention mechanism, that efficiently identifies input regions for which the DCN's output is most sensitive and to which we should devote more capacity. We focus our empirical evaluation on the Cluttered MNIST and SVHN image datasets. Our findings indicate that DCNs are able to drastically reduce the number of computations, compared to traditional convolutional neural networks, while maintaining similar or even better performance.
2016-06-10
Proceedings of The 33rd International Conference on Machine Learning (publié)
doi.org
proceedings.mlr.press