DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset

A large diverse collection of expert demonstrations that are used to train foundational models for robot manipulation. 

Engineers working with medical robotic equipment.


While Large Language Models (LLMs) have excelled in natural language processing tasks thanks to extensive datasets, robotics faces a challenge in this regard, particularly in manipulation tasks. 

Unlike LLMs, which benefit from vast text corpora, robotics lacks comparably large datasets for manipulation tasks. 

From early 2023, several members of the Robotics and Embodied AI (REAL) Lab have collaborated with 12 other research labs around the world to produce a large, diverse, high-quality robot manipulation dataset. 


The team at REAL recruited data collectors from the student population at Mila and Université de Montréal and produced thousands of trajectories of household manipulation tasks, with the goal to leverage this dataset to work toward more capable and robust robotic manipulation policies. 

This initiative aims to address the scarcity of comprehensive datasets in robotics, enhancing the development of manipulation algorithms for real-world applications.

Each lab built identical teleoperation systems to enable consistent data collection across all institutions. 

A Franka robot arm was mounted on a height-adjustable rolling desk, which enabled us to wheel the setup to different scenes and diversify the data. 


The setup was equipped with three stereo cameras, two Zed 2 cameras providing adjustable over-the-shoulder views, and a wristmounted Zed Mini, giving a detailed gripper view. 


The adaptable setup enables diverse data, from scenes to multiple robot positions to varied camera positions. As an example, the setup was wheeled to the MILA kitchen, and coffee making data was collected.

Photo of Glen Berseth.

Recently, the machine learning community has been learning the importance of data diversity. In this work, we focus on collecting robot manipulation data with a larger task and visual diversity. This greatly increases the ability for large models trained for robotics.

Glen Berseth, Assistant Professor, Université de Montréal, Core Academic Member, Mila


DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
DROID dataset
Full dataset, policy learning code, and a detailed guide for reproducing our robot hardware setup.

Meet the Team

Mila Members
Core Academic Member
Portrait of Glen Berseth
Assistant Professor, Université de Montréal, Department of Computer Science and Operations Research
Canada CIFAR AI Chair
Portrait of Kirsty Ellis is unavailable
Developer, Research Software, Innovation, Development and Technologies