DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset
A large diverse collection of expert demonstrations that are used to train foundational models for robot manipulation.
Background
While Large Language Models (LLMs) have excelled in natural language processing tasks thanks to extensive datasets, robotics faces a challenge in this regard, particularly in manipulation tasks.
Unlike LLMs, which benefit from vast text corpora, robotics lacks comparably large datasets for manipulation tasks.
From early 2023, several members of the Robotics and Embodied AI (REAL) Lab have collaborated with 12 other research labs around the world to produce a large, diverse, high-quality robot manipulation dataset.
Objectives
The team at REAL recruited data collectors from the student population at Mila and Université de Montréal and produced thousands of trajectories of household manipulation tasks, with the goal to leverage this dataset to work toward more capable and robust robotic manipulation policies.
This initiative aims to address the scarcity of comprehensive datasets in robotics, enhancing the development of manipulation algorithms for real-world applications.
Each lab built identical teleoperation systems to enable consistent data collection across all institutions.
A Franka robot arm was mounted on a height-adjustable rolling desk, which enabled us to wheel the setup to different scenes and diversify the data.
The setup was equipped with three stereo cameras, two Zed 2 cameras providing adjustable over-the-shoulder views, and a wristmounted Zed Mini, giving a detailed gripper view.
The adaptable setup enables diverse data, from scenes to multiple robot positions to varied camera positions. As an example, the setup was wheeled to the MILA kitchen, and coffee making data was collected.
Recently, the machine learning community has been learning the importance of data diversity. In this work, we focus on collecting robot manipulation data with a larger task and visual diversity. This greatly increases the ability for large models trained for robotics.
Glen Berseth, Assistant Professor, Université de Montréal, Core Academic Member, Mila
Resources
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
Read the paper
https://arxiv.org/abs/2403.12945
DROID dataset
Full dataset, policy learning code, and a detailed guide for reproducing our robot hardware setup.
Learn more
https://droid-dataset.github.io/
Meet the Team
Mila Members
Partners
