Florian Golemo

Alexia Jolicoeur-Martineau

Liam Paull

Video diffusion techniques have advanced significantly in recent years; however, they struggle to generate realistic imagery of car crashes … (voir plus)due to the scarcity of accident events in most driving datasets. Improving traffic safety requires realistic and controllable accident simulations. To tackle the problem, we propose Ctrl-Crash, a controllable car crash video generation model that conditions on signals such as bounding boxes, crash types, and an initial image frame. Our approach enables counterfactual scenario generation where minor variations in input can lead to dramatically different crash outcomes. To support fine-grained control at inference time, we leverage classifier-free guidance with independently tunable scales for each conditioning signal. Ctrl-Crash achieves state-of-the-art performance across quantitative video quality metrics (e.g., FVD and JEDi) and qualitative measurements based on a human-evaluation of physical realism and video quality compared to prior diffusion-based methods.

2025-06-01

arXiv (publié)

Ctrl-Crash: Controllable Diffusion for Realistic Car Crashes

Anthony Gosselin

Ge Ya Luo

Luis Lara

Alexia Jolicoeur-Martineau

Liam Paull

Video diffusion techniques have advanced significantly in recent years; however, they struggle to generate realistic imagery of car crashes … (voir plus)due to the scarcity of accident events in most driving datasets. Improving traffic safety requires realistic and controllable accident simulations. To tackle the problem, we propose Ctrl-Crash, a controllable car crash video generation model that conditions on signals such as bounding boxes, crash types, and an initial image frame. Our approach enables counterfactual scenario generation where minor variations in input can lead to dramatically different crash outcomes. To support fine-grained control at inference time, we leverage classifier-free guidance with independently tunable scales for each conditioning signal. Ctrl-Crash achieves state-of-the-art performance across quantitative video quality metrics (e.g., FVD and JEDi) and qualitative measurements based on a human-evaluation of physical realism and video quality compared to prior diffusion-based methods.

2025-05-30

ArXiv (prépublication)

CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning

Felix Heide

Evaluating autonomous vehicle stacks (AVs) in simulation typically involves replaying driving logs from real-world recorded traffic. However… (voir plus), agents replayed from offline data do not react to the actions of the AV, and their behaviour cannot be easily controlled to simulate counterfactual scenarios. Existing approaches have attempted to address these shortcomings by proposing methods that rely on heuristics or learned generative models of real-world data but these approaches either lack realism or necessitate costly iterative sampling procedures to control the generated behaviours. In this work, we take an alternative approach and propose CtRL-Sim, a method that leverages return-conditioned offline reinforcement learning within a physics-enhanced Nocturne simulator to efficiently generate reactive and controllable traffic agents. Specifically, we process real-world driving data through the Nocturne simulator to generate a diverse offline reinforcement learning dataset, annotated with various reward terms. With this dataset, we train a return-conditioned multi-agent behaviour model that allows for fine-grained manipulation of agent behaviours by modifying the desired returns for the various reward components. This capability enables the generation of a wide range of driving behaviours beyond the scope of the initial dataset, including those representing adversarial behaviours. We demonstrate that CtRL-Sim can efficiently generate diverse and realistic safety-critical scenarios while providing fine-grained control over agent behaviours. Further, we show that fine-tuning our model on simulated safety-critical scenarios generated by our model enhances this controllability.

2024-09-05

robot-learning.org/CoRL/2024/Conference (accepté)

CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning

Felix Heide

Evaluating autonomous vehicle stacks (AVs) in simulation typically involves replaying driving logs from real-world recorded traffic. However… (voir plus), agents replayed from offline data are not reactive and hard to intuitively control. Existing approaches address these challenges by proposing methods that rely on heuristics or generative models of real-world data but these approaches either lack realism or necessitate costly iterative sampling procedures to control the generated behaviours. In this work, we take an alternative approach and propose CtRL-Sim, a method that leverages return-conditioned offline reinforcement learning (RL) to efficiently generate reactive and controllable traffic agents. Specifically, we process real-world driving data through a physics-enhanced Nocturne simulator to generate a diverse offline RL dataset, annotated with various rewards. With this dataset, we train a return-conditioned multi-agent behaviour model that allows for fine-grained manipulation of agent behaviours by modifying the desired returns for the various reward components. This capability enables the generation of a wide range of driving behaviours beyond the scope of the initial dataset, including adversarial behaviours. We show that CtRL-Sim can generate realistic safety-critical scenarios while providing fine-grained control over agent behaviours.

2024-03-29

ArXiv (prépublication)

GrowSpace: A reinforcement learning environment for plant architecture

Yasmeen Hitti

Ionelia Buzatu

Manuel Del Verme

Mark Lefsrud

Audrey Durand

2024-02-01

Computers and Electronics in Agriculture (publié)

Visual Question Answering From Another Perspective: CLEVR Mental Rotation Tests

Christopher Beckham

Sina Honari

2023-04-01

Pattern Recognition (publié)

Towards Learning to Imitate from a Single Video Demonstration

Glen Berseth

Agents that can learn to imitate given video observation -- \emph{without direct access to state or action information} are more applicable … (voir plus)to learning in the natural world. However, formulating a reinforcement learning (RL) agent that facilitates this goal remains a significant challenge. We approach this challenge using contrastive training to learn a reward function comparing an agent's behaviour with a single demonstration. We use a Siamese recurrent neural network architecture to learn rewards in space and time between motion clips while training an RL policy to minimize this distance. Through experimentation, we also find that the inclusion of multi-task data and additional image encoding losses improve the temporal consistency of the learned rewards and, as a result, significantly improves policy learning. We demonstrate our approach on simulated humanoid, dog, and raptor agents in 2D and a quadruped and a humanoid in 3D. We show that our method outperforms current state-of-the-art techniques in these environments and can learn to imitate from a single video demonstration.

Kubric: A scalable dataset generator

Klaus Greff

Francois Belletti

Lucas Beyer

Carl Doersch

Yilun Du

Daniel Duckworth

David J Fleet

Dan Gnanapragasam

Charles Herrmann

Thomas Kipf

Abhijit Kundu

Dmitry Lagun

Issam Hadj Laradji

Hsueh-Ti Liu

Henning Meyer

Yishu Miao

Cengiz Oztireli

Etienne Pot … (voir 14 de plus)

Noha Radwan

Daniel Rebain

Sara Sabour

Mehdi S. M. Sajjadi

Matan Sela

Vincent Sitzmann

Austin Stone

Deqing Sun

Suhani Vora

Ziyu Wang

Tianhao Wu

Kwang Moo Yi

Fangcheng Zhong

Andrea Tagliasacchi

Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance o… (voir plus)f a system than architecture and training details. But collecting, processing and annotating real data at scale is difficult, expensive, and frequently raises additional privacy, fairness and legal concerns. Synthetic data is a powerful tool with the potential to address these shortcomings: 1) it is cheap 2) supports rich ground-truth annotations 3) offers full control over data and 4) can circumvent or mitigate problems regarding bias, privacy and licensing. Unfortunately, software tools for effective data generation are less mature than those for architecture design and training, which leads to fragmented generation efforts. To address these problems we introduce Kubric, an open-source Python framework that interfaces with PyBullet and Blender to generate photo-realistic scenes, with rich annotations, and seamlessly scales to large jobs distributed over thousands of machines, and generating TBs of data. We demonstrate the effectiveness of Kubric by presenting a series of 13 different generated datasets for tasks ranging from studying 3D NeRF models to optical flow estimation. We release Kubric, the used assets, all of the generation code, as well as the rendered datasets for reuse and modification.

2022-06-18

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (publié)

Kubric: A scalable dataset generator

Klaus Greff

Francois Belletti

Lucas Beyer

Carl Doersch

Yilun Du

Daniel Duckworth

David J. Fleet

Dan Gnanapragasam

Charles Herrmann

Thomas N. Kipf

Abhijit Kundu

Dmitry Lagun

Issam Hadj Laradji

Hsueh-Ti Liu

H. Meyer

Yishu Miao

Cengiz Oztireli

Etienne Pot … (voir 14 de plus)

Noha Radwan

Daniel Rebain

Sara Sabour

Mehdi S. M. Sajjadi

Matan Sela

Vincent Sitzmann

Austin Stone

Deqing Sun

Suhani Vora

Ziyu Wang

Tianhao Wu

Kwang Moo Yi

Fangcheng Zhong

Andrea Tagliasacchi

Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance o… (voir plus)f a system than architecture and training details. But collecting, processing and annotating real data at scale is difficult, expensive, and frequently raises additional privacy, fairness and legal concerns. Synthetic data is a powerful tool with the potential to address these shortcomings: 1) it is cheap 2) supports rich ground-truth annotations 3) offers full control over data and 4) can circumvent or mitigate problems regarding bias, privacy and licensing. Unfortunately, software tools for effective data generation are less mature than those for architecture design and training, which leads to fragmented generation efforts. To address these problems we introduce Kubric, an open-source Python framework that interfaces with PyBullet and Blender to generate photo-realistic scenes, with rich annotations, and seamlessly scales to large jobs distributed over thousands of machines, and generating TBs of data. We demonstrate the effectiveness of Kubric by presenting a series of 13 different generated datasets for tasks ranging from studying 3D NeRF models to optical flow estimation. We release Kubric, the used assets, all of the generation code, as well as the rendered datasets for reuse and modification.

2022-03-07

ArXiv (preprint)

GrowSpace: Learning How to Shape Plants

Yasmeen Hitti

Ionelia Buzatu

Manuel Del Verme

Mark Lefsrud

Audrey Durand

Plants are dynamic systems that are integral to our existence and survival. Plants face environment changes and adapt over time to their sur… (voir plus)rounding conditions. We argue that plant responses to an environmental stimulus are a good example of a real-world problem that can be approached within a reinforcement learning (RL)framework. With the objective of controlling a plant by moving the light source, we propose GrowSpace, as a new RL benchmark. The back-end of the simulator is implemented using the Space Colonisation Algorithm, a plant growing model based on competition for space. Compared to video game RL environments, this simulator addresses a real-world problem and serves as a test bed to visualize plant growth and movement in a faster way than physical experiments. GrowSpace is composed of a suite of challenges that tackle several problems such as control, multi-stage learning,fairness and multi-objective learning. We provide agent baselines alongside case studies to demonstrate the difficulty of the proposed benchmark.

2022-01-01

AAAI.org/2022/Workshop/AIAFS (publié)

Latent Variable Sequential Set Transformers for Joint Multi-Agent Motion Prediction

Roger Girgis

Felipe Codevilla

Martin Weiss

Jim Aldon D'Souza

Samira Ebrahimi Kahou

Felix Heide

Robust multi-agent trajectory prediction is essential for the safe control of robotic systems. A major challenge is to efficiently learn a r… (voir plus)epresentation that approximates the true joint distribution of contextual, social, and temporal information to enable planning. We propose Latent Variable Sequential Set Transformers which are encoder-decoder architectures that generate scene-consistent multi-agent trajectories. We refer to these architectures as “AutoBots”. The encoder is a stack of interleaved temporal and social multi-head self-attention (MHSA) modules which alternately perform equivariant processing across the temporal and social dimensions. The decoder employs learnable seed parameters in combination with temporal and social MHSA modules allowing it to perform inference over the entire future scene in a single forward pass efficiently. AutoBots can produce either the trajectory of one ego-agent or a distribution over the future trajectories for all agents in the scene. For the single-agent prediction case, our model achieves top results on the global nuScenes vehicle motion prediction leaderboard, and produces strong results on the Argoverse vehicle prediction challenge. In the multi-agent setting, we evaluate on the synthetic partition of TrajNet++ dataset to showcase the model’s socially-consistent predictions. We also demonstrate our model on general sequences of sets and provide illustrative experiments modelling the sequential structure of the multiple strokes that make up symbols in the Omniglot data. A distinguishing feature of AutoBots is that all models are trainable on a single desktop GPU (1080 Ti) in under 48h.

2022-01-01

ICLR (publié)

Haptics-based Curiosity for Sparse-reward Tasks

Sai Rajeswar

Cyril Ibrahim

Nitin Surya

David Vazquez

Aaron Courville

Pedro O. Pinheiro

Robots in many real-world settings have access to force/torque sensors in their gripper and tactile sensing is often necessary for tasks tha… (voir plus)t involve contact-rich motion. In this work, we leverage surprise from mismatches in haptics feedback to guide exploration in hard sparse-reward reinforcement learning tasks. Our approach, Haptics-based Curiosity (\method{}), learns what visible objects interactions are supposed to ``feel" like. We encourage exploration by rewarding interactions where the expectation and the experience do not match. We test our approach on a range of haptics-intensive robot arm tasks (e.g. pushing objects, opening doors), which we also release as part of this work. Across multiple experiments in a simulated setting, we demonstrate that our method is able to learn these difficult tasks through sparse reward and curiosity alone. We compare our cross-modal approach to single-modality (haptics- or vision-only) approaches as well as other curiosity-based methods and find that our method performs better and is more sample-efficient.

2021-09-13

robot-learning.org/CoRL/2021/Conference (poster)

proceedings.mlr.press