Glen Berseth

Nikolay Malkin

Diffusion models have emerged as effective distribution estimators in vision, language, and reinforcement learning, but their use as priors … (voir plus)in downstream tasks poses an intractable posterior inference problem. This paper studies amortized sampling of the posterior over data,

2024-05-31

ArXiv (prépublication)

Amortizing intractable inference in diffusion models for vision, language, and control

Siddarth Venkatraman

Moksh J. Jain

Luca Scimeca

Minsu Kim

Marcin Sendera

Mohsin Hasan

Luke Rowe

Sarthak Mittal

Pablo Lemos

Emmanuel Bengio

Alexandre Adam

Jarrid Rector-Brooks

Yoshua Bengio

Nikolay Malkin

Diffusion models have emerged as effective distribution estimators in vision, language, and reinforcement learning, but their use as priors … (voir plus)in downstream tasks poses an intractable posterior inference problem. This paper studies amortized sampling of the posterior over data,

2024-05-31

ArXiv (prépublication)

Surprise-Adaptive Intrinsic Motivation for Unsupervised Reinforcement Learning

Adriana Hugessen

Roger Creus Castanyer

Faisal Mohamed

Both entropy-minimizing and entropy-maximizing (curiosity) objectives for unsupervised reinforcement learning (RL) have been shown to be eff… (voir plus)ective in different environments, depending on the environment's level of natural entropy. However, neither method alone results in an agent that will consistently learn intelligent behavior across environments. In an effort to find a single entropy-based method that will encourage emergent behaviors in any environment, we propose an agent that can adapt its objective online, depending on the entropy conditions by framing the choice as a multi-armed bandit problem. We devise a novel intrinsic feedback signal for the bandit, which captures the agent's ability to control the entropy in its environment. We demonstrate that such agents can learn to control entropy and exhibit emergent behaviors in both high- and low-entropy regimes and can learn skillful behaviors in benchmark tasks. Videos of the trained agents and summarized findings can be found on our project page https://sites.google.com/view/surprise-adaptive-agents

2024-05-14

rl-conference.cc/RLC/2024/Conference (publié)

openreview.net

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

Alexander Khazatsky

Karl Pertsch

Suraj Nair

Ashwin Balakrishna

Sudeep Dasari

Siddharth Karamcheti

Soroush Nasiriany

Mohan Kumar Srirama

Lawrence Yunliang Chen

Kirsty Ellis

Peter David Fagan

Joey Hejna

Masha Itkina

Marion Lepert

Ye Ma

Patrick Tree Miller

Jimmy Wu

Suneel Belkhale

Shivin Dass

Huy Ha … (voir 79 de plus)

Arhan Jain

Abraham Lee

Youngwoon Lee

Marius Memmel

Sungjae Park

Ilija Radosavovic

Kaiyuan Wang

Albert Zhan

Kevin Black

Cheng Chi

Kyle Beltran Hatch

Shan Lin

Jingpei Lu

Jean Mercat

Abdul Rehman

Pannag R Sanketi

Archit Sharma

Cody Simpson

Quan Vuong

Homer Rich Walke

Blake Wulfe

Ted Xiao

Jonathan Heewon Yang

Arefeh Yavary

Tony Z. Zhao

Christopher Agia

Rohan Baijal

Mateo Guaman Castro

Daphne Chen

Qiuyu Chen

Trinity Chung

Jaimyn Drake

Ethan Paul Foster

Jensen Gao

David Antonio Herrera

Minho Heo

Kyle Hsu

Jiaheng Hu

Donovon Jackson

Charlotte Le

Yunshuang Li

K. Lin

Roy Lin

Zehan Ma

Abhiram Maddukuri

Suvir Mirchandani

Daniel Morton

Tony Khuong Nguyen

Abigail O'Neill

Rosario Scalise

Derick Seale

Victor Son

Stephen Tian

Emi Tran

Andrew E. Wang

Yilin Wu

Annie Xie

Jingyun Yang

Patrick Yin

Yunchu Zhang

Osbert Bastani

Jeannette Bohg

Ken Goldberg

Abhinav Gupta

Abhishek Gupta

Dinesh Jayaraman

Joseph J Lim

Jitendra Malik

Roberto Mart'in-Mart'in

Subramanian Ramamoorthy

Dorsa Sadigh

Shuran Song

Jiajun Wu

Michael C. Yip

Yuke Zhu

Thomas Kollar

Sergey Levine

Chelsea Finn

The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and … (voir plus)robust robotic manipulation policies. However, creating such datasets is challenging: collecting robot manipulation data in diverse environments poses logistical and safety challenges and requires substantial investments in hardware and human labour. As a result, even the most general robot manipulation policies today are mostly trained on data collected in a small number of environments with limited scene and task diversity. In this work, we introduce DROID (Distributed Robot Interaction Dataset), a diverse robot manipulation dataset with 76k demonstration trajectories or 350 hours of interaction data, collected across 564 scenes and 84 tasks by 50 data collectors in North America, Asia, and Europe over the course of 12 months. We demonstrate that training with DROID leads to policies with higher performance and improved generalization ability. We open source the full dataset, policy learning code, and a detailed guide for reproducing our robot hardware setup.

2024-03-19

ArXiv (prépublication)

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

Alexander Khazatsky

Karl Pertsch

Suraj Nair

Ashwin Balakrishna

Sudeep Dasari

Siddharth Karamcheti

Soroush Nasiriany

Mohan Kumar Srirama

Lawrence Yunliang Chen

Kirsty Ellis

Peter David Fagan

Joey Hejna

Masha Itkina

Marion Lepert

Ye Ma

Patrick Tree Miller

Jimmy Wu

Suneel Belkhale

Shivin Dass

Huy Ha … (voir 79 de plus)

Arhan Jain

Abraham Lee

Youngwoon Lee

Marius Memmel

Sungjae Park

Ilija Radosavovic

Kaiyuan Wang

Albert Zhan

Kevin Black

Cheng Chi

Kyle Beltran Hatch

Shan Lin

Jingpei Lu

Jean Mercat

Abdul Rehman

Pannag R Sanketi

Archit Sharma

Cody Simpson

Quan Vuong

Homer Rich Walke

Blake Wulfe

Ted Xiao

Jonathan Heewon Yang

Arefeh Yavary

Tony Z. Zhao

Christopher Agia

Rohan Baijal

Mateo Guaman Castro

Daphne Chen

Qiuyu Chen

Trinity Chung

Jaimyn Drake

Ethan Paul Foster

Jensen Gao

David Antonio Herrera

Minho Heo

Kyle Hsu

Jiaheng Hu

Donovon Jackson

Charlotte Le

Yunshuang Li

K. Lin

Roy Lin

Zehan Ma

Abhiram Maddukuri

Suvir Mirchandani

Daniel Morton

Tony Khuong Nguyen

Abigail O'Neill

Rosario Scalise

Derick Seale

Victor Son

Stephen Tian

Emi Tran

Andrew E. Wang

Yilin Wu

Annie Xie

Jingyun Yang

Patrick Yin

Yunchu Zhang

Osbert Bastani

Jeannette Bohg

Ken Goldberg

Abhinav Gupta

Abhishek Gupta

Dinesh Jayaraman

Joseph J Lim

Jitendra Malik

Roberto Mart'in-Mart'in

Subramanian Ramamoorthy

Dorsa Sadigh

Shuran Song

Jiajun Wu

Michael C. Yip

Yuke Zhu

Thomas Kollar

Sergey Levine

Chelsea Finn

The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and … (voir plus)robust robotic manipulation policies. However, creating such datasets is challenging: collecting robot manipulation data in diverse environments poses logistical and safety challenges and requires substantial investments in hardware and human labour. As a result, even the most general robot manipulation policies today are mostly trained on data collected in a small number of environments with limited scene and task diversity. In this work, we introduce DROID (Distributed Robot Interaction Dataset), a diverse robot manipulation dataset with 76k demonstration trajectories or 350 hours of interaction data, collected across 564 scenes and 84 tasks by 50 data collectors in North America, Asia, and Europe over the course of 12 months. We demonstrate that training with DROID leads to policies with higher performance and improved generalization ability. We open source the full dataset, policy learning code, and a detailed guide for reproducing our robot hardware setup.

2024-03-19

ArXiv (prépublication)

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

Alexander Khazatsky

Karl Pertsch

Suraj Nair

Ashwin Balakrishna

Sudeep Dasari

Siddharth Karamcheti

Soroush Nasiriany

Mohan Kumar Srirama

Lawrence Yunliang Chen

Kirsty Ellis

Peter David Fagan

Joey Hejna

Masha Itkina

Marion Lepert

Ye Ma

Patrick Tree Miller

Jimmy Wu

Suneel Belkhale

Shivin Dass

Huy Ha … (voir 79 de plus)

Arhan Jain

Abraham Lee

Youngwoon Lee

Marius Memmel

Sungjae Park

Ilija Radosavovic

Kaiyuan Wang

Albert Zhan

Kevin Black

Cheng Chi

Kyle Beltran Hatch

Shan Lin

Jingpei Lu

Jean Mercat

Abdul Rehman

Pannag R Sanketi

Archit Sharma

Cody Simpson

Quan Vuong

Homer Rich Walke

Blake Wulfe

Ted Xiao

Jonathan Heewon Yang

Arefeh Yavary

Tony Z. Zhao

Christopher Agia

Rohan Baijal

Mateo Guaman Castro

Daphne Chen

Qiuyu Chen

Trinity Chung

Jaimyn Drake

Ethan Paul Foster

Jensen Gao

David Antonio Herrera

Minho Heo

Kyle Hsu

Jiaheng Hu

Donovon Jackson

Charlotte Le

Yunshuang Li

K. Lin

Roy Lin

Zehan Ma

Abhiram Maddukuri

Suvir Mirchandani

Daniel Morton

Tony Khuong Nguyen

Abigail O'Neill

Rosario Scalise

Derick Seale

Victor Son

Stephen Tian

Emi Tran

Andrew E. Wang

Yilin Wu

Annie Xie

Jingyun Yang

Patrick Yin

Yunchu Zhang

Osbert Bastani

Jeannette Bohg

Ken Goldberg

Abhinav Gupta

Abhishek Gupta

Dinesh Jayaraman

Joseph J Lim

Jitendra Malik

Roberto Mart'in-Mart'in

Subramanian Ramamoorthy

Dorsa Sadigh

Shuran Song

Jiajun Wu

Michael C. Yip

Yuke Zhu

Thomas Kollar

Sergey Levine

Chelsea Finn

The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and … (voir plus)robust robotic manipulation policies. However, creating such datasets is challenging: collecting robot manipulation data in diverse environments poses logistical and safety challenges and requires substantial investments in hardware and human labour. As a result, even the most general robot manipulation policies today are mostly trained on data collected in a small number of environments with limited scene and task diversity. In this work, we introduce DROID (Distributed Robot Interaction Dataset), a diverse robot manipulation dataset with 76k demonstration trajectories or 350 hours of interaction data, collected across 564 scenes and 84 tasks by 50 data collectors in North America, Asia, and Europe over the course of 12 months. We demonstrate that training with DROID leads to policies with higher performance and improved generalization ability. We open source the full dataset, policy learning code, and a detailed guide for reproducing our robot hardware setup.

2024-03-19

ArXiv (prépublication)

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

Alexander Khazatsky

Karl Pertsch

Suraj Nair

Ashwin Balakrishna

Sudeep Dasari

Siddharth Karamcheti

Soroush Nasiriany

Mohan Kumar Srirama

Lawrence Yunliang Chen

Kirsty Ellis

Peter David Fagan

Joey Hejna

Masha Itkina

Marion Lepert

Ye Ma

Patrick Tree Miller

Jimmy Wu

Suneel Belkhale

Shivin Dass

Huy Ha … (voir 79 de plus)

Arhan Jain

Abraham Lee

Youngwoon Lee

Marius Memmel

Sungjae Park

Ilija Radosavovic

Kaiyuan Wang

Albert Zhan

Kevin Black

Cheng Chi

Kyle Beltran Hatch

Shan Lin

Jingpei Lu

Jean Mercat

Abdul Rehman

Pannag R Sanketi

Archit Sharma

Cody Simpson

Quan Vuong

Homer Rich Walke

Blake Wulfe

Ted Xiao

Jonathan Heewon Yang

Arefeh Yavary

Tony Z. Zhao

Christopher Agia

Rohan Baijal

Mateo Guaman Castro

Daphne Chen

Qiuyu Chen

Trinity Chung

Jaimyn Drake

Ethan Paul Foster

Jensen Gao

David Antonio Herrera

Minho Heo

Kyle Hsu

Jiaheng Hu

Donovon Jackson

Charlotte Le

Yunshuang Li

K. Lin

Roy Lin

Zehan Ma

Abhiram Maddukuri

Suvir Mirchandani

Daniel Morton

Tony Khuong Nguyen

Abigail O'Neill

Rosario Scalise

Derick Seale

Victor Son

Stephen Tian

Emi Tran

Andrew E. Wang

Yilin Wu

Annie Xie

Jingyun Yang

Patrick Yin

Yunchu Zhang

Osbert Bastani

Jeannette Bohg

Ken Goldberg

Abhinav Gupta

Abhishek Gupta

Dinesh Jayaraman

Joseph J Lim

Jitendra Malik

Roberto Mart'in-Mart'in

Subramanian Ramamoorthy

Dorsa Sadigh

Shuran Song

Jiajun Wu

Michael C. Yip

Yuke Zhu

Thomas Kollar

Sergey Levine

Chelsea Finn

The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and … (voir plus)robust robotic manipulation policies. However, creating such datasets is challenging: collecting robot manipulation data in diverse environments poses logistical and safety challenges and requires substantial investments in hardware and human labour. As a result, even the most general robot manipulation policies today are mostly trained on data collected in a small number of environments with limited scene and task diversity. In this work, we introduce DROID (Distributed Robot Interaction Dataset), a diverse robot manipulation dataset with 76k demonstration trajectories or 350 hours of interaction data, collected across 564 scenes and 84 tasks by 50 data collectors in North America, Asia, and Europe over the course of 12 months. We demonstrate that training with DROID leads to policies with higher performance and improved generalization ability. We open source the full dataset, policy learning code, and a detailed guide for reproducing our robot hardware setup.

2024-03-19

ArXiv (prépublication)

Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control

Zhongyu Li

Xue Bin Peng

Pieter Abbeel

Sergey Levine

Koushil Sreenath

This paper presents a comprehensive study on using deep reinforcement learning (RL) to create dynamic locomotion controllers for bipedal rob… (voir plus)ots. Going beyond focusing on a single locomotion skill, we develop a general control solution that can be used for a range of dynamic bipedal skills, from periodic walking and running to aperiodic jumping and standing. Our RL-based controller incorporates a novel dual-history architecture, utilizing both a long-term and short-term input/output (I/O) history of the robot. This control architecture, when trained through the proposed end-to-end RL approach, consistently outperforms other methods across a diverse range of skills in both simulation and the real world. The study also delves into the adaptivity and robustness introduced by the proposed RL system in developing locomotion controllers. We demonstrate that the proposed architecture can adapt to both time-invariant dynamics shifts and time-variant changes, such as contact events, by effectively using the robot’s I/O history. Additionally, we identify task randomization as another key source of robustness, fostering better task generalization and compliance to disturbances. The resulting control policies can be successfully deployed on Cassie, a torque-controlled human-sized bipedal robot. This work pushes the limits of agility for bipedal robots through extensive real-world experiments. We demonstrate a diverse range of locomotion skills, including: robust standing, versatile walking, fast running with a demonstration of a 400-meter dash, and a diverse set of jumping skills, such as standing long jumps and high jumps.

2024-01-30

ArXiv (prépublication)

Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control

Zhongyu Li

Xue Bin Peng

Pieter Abbeel

Sergey Levine

Koushil Sreenath

This paper presents a comprehensive study on using deep reinforcement learning (RL) to create dynamic locomotion controllers for bipedal rob… (voir plus)ots. Going beyond focusing on a single locomotion skill, we develop a general control solution that can be used for a range of dynamic bipedal skills, from periodic walking and running to aperiodic jumping and standing. Our RL-based controller incorporates a novel dual-history architecture, utilizing both a long-term and short-term input/output (I/O) history of the robot. This control architecture, when trained through the proposed end-to-end RL approach, consistently outperforms other methods across a diverse range of skills in both simulation and the real world. The study also delves into the adaptivity and robustness introduced by the proposed RL system in developing locomotion controllers. We demonstrate that the proposed architecture can adapt to both time-invariant dynamics shifts and time-variant changes, such as contact events, by effectively using the robot’s I/O history. Additionally, we identify task randomization as another key source of robustness, fostering better task generalization and compliance to disturbances. The resulting control policies can be successfully deployed on Cassie, a torque-controlled human-sized bipedal robot. This work pushes the limits of agility for bipedal robots through extensive real-world experiments. We demonstrate a diverse range of locomotion skills, including: robust standing, versatile walking, fast running with a demonstration of a 400-meter dash, and a diverse set of jumping skills, such as standing long jumps and high jumps.

2024-01-30

ArXiv (prépublication)

Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control

Zhongyu Li

Xue Bin Peng

Pieter Abbeel

Sergey Levine

Koushil Sreenath

This paper presents a comprehensive study on using deep reinforcement learning (RL) to create dynamic locomotion controllers for bipedal rob… (voir plus)ots. Going beyond focusing on a single locomotion skill, we develop a general control solution that can be used for a range of dynamic bipedal skills, from periodic walking and running to aperiodic jumping and standing. Our RL-based controller incorporates a novel dual-history architecture, utilizing both a long-term and short-term input/output (I/O) history of the robot. This control architecture, when trained through the proposed end-to-end RL approach, consistently outperforms other methods across a diverse range of skills in both simulation and the real world. The study also delves into the adaptivity and robustness introduced by the proposed RL system in developing locomotion controllers. We demonstrate that the proposed architecture can adapt to both time-invariant dynamics shifts and time-variant changes, such as contact events, by effectively using the robot’s I/O history. Additionally, we identify task randomization as another key source of robustness, fostering better task generalization and compliance to disturbances. The resulting control policies can be successfully deployed on Cassie, a torque-controlled human-sized bipedal robot. This work pushes the limits of agility for bipedal robots through extensive real-world experiments. We demonstrate a diverse range of locomotion skills, including: robust standing, versatile walking, fast running with a demonstration of a 400-meter dash, and a diverse set of jumping skills, such as standing long jumps and high jumps.

2024-01-30

ArXiv (prépublication)

Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control

Zhongyu Li

Xue Bin Peng

Pieter Abbeel

Sergey Levine

Koushil Sreenath

This paper presents a comprehensive study on using deep reinforcement learning (RL) to create dynamic locomotion controllers for bipedal rob… (voir plus)ots. Going beyond focusing on a single locomotion skill, we develop a general control solution that can be used for a range of dynamic bipedal skills, from periodic walking and running to aperiodic jumping and standing. Our RL-based controller incorporates a novel dual-history architecture, utilizing both a long-term and short-term input/output (I/O) history of the robot. This control architecture, when trained through the proposed end-to-end RL approach, consistently outperforms other methods across a diverse range of skills in both simulation and the real world. The study also delves into the adaptivity and robustness introduced by the proposed RL system in developing locomotion controllers. We demonstrate that the proposed architecture can adapt to both time-invariant dynamics shifts and time-variant changes, such as contact events, by effectively using the robot’s I/O history. Additionally, we identify task randomization as another key source of robustness, fostering better task generalization and compliance to disturbances. The resulting control policies can be successfully deployed on Cassie, a torque-controlled human-sized bipedal robot. This work pushes the limits of agility for bipedal robots through extensive real-world experiments. We demonstrate a diverse range of locomotion skills, including: robust standing, versatile walking, fast running with a demonstration of a 400-meter dash, and a diverse set of jumping skills, such as standing long jumps and high jumps.

2024-01-30

ArXiv (prépublication)

Closing the Gap between TD Learning and Supervised Learning - A Generalisation Point of View

Raj Ghugare

Matthieu Geist

Benjamin Eysenbach

Some reinforcement learning (RL) algorithms can stitch pieces of experience to solve a task never seen before during training. This oft-soug… (voir plus)ht property is one of the few ways in which RL methods based on dynamic-programming differ from RL methods based on supervised-learning (SL). Yet, certain RL methods based on off-the-shelf SL algorithms achieve excellent results without an explicit mechanism for stitching; it remains unclear whether those methods forgo this important stitching property. This paper studies this question for the problems of achieving a target goal state and achieving a target return value. Our main result is to show that the stitching property corresponds to a form of combinatorial generalization: after training on a distribution of (state, goal) pairs, one would like to evaluate on (state, goal) pairs not seen together in the training data. Our analysis shows that this sort of generalization is different from i.i.d. generalization. This connection between stitching and generalisation reveals why we should not expect SL-based RL methods to perform stitching, even in the limit of large datasets and models. Based on this analysis, we construct new datasets to explicitly test for this property, revealing that SL-based methods lack this stitching property and hence fail to perform combinatorial generalization. Nonetheless, the connection between stitching and combinatorial generalisation also suggests a simple remedy for improving generalisation in SL: data augmentation. We propose a temporal data augmentation and demonstrate that adding it to SL-based methods enables them to successfully complete tasks not seen together during training. On a high level, this connection illustrates the importance of combinatorial generalization for data efficiency in time-series data beyond tasks beyond RL, like audio, video, or text.

2024-01-16

ICLR.cc/2024/Conference (poster)