What is teleoperation data?

Recordings of a person driving a robot through a task: camera frames, joint states, forces and the commands sent, synchronized in time. Robot learning models use them as demonstrations.

Which format should I use: LeRobot, RLDS or HDF5?

Use what your training code reads. LeRobot suits Hugging Face and PyTorch workflows, RLDS suits TensorFlow and cross-embodiment pipelines, and HDF5 is the simple, universal option.

How many demonstrations does a task need?

Research systems report tens to hundreds per task for single-task policies. Collect a pilot batch, evaluate, and collect more where the policy fails.

Should failed episodes be kept?

Keep them, but label them. They help with evaluation and some training methods, and they should never sit unlabeled among successful demonstrations.

Guide

How to collect teleoperation data for robot learning (LeRobot, RLDS, HDF5)

Policies learn what your demonstrations show them. A consistent protocol matters more than a clever interface.

Illustration

Updated October 9, 2026

Imitation learning and vision-language-action models are trained on demonstrations: a person drives the robot through a task while every camera frame, joint position and command is recorded. The quality of that data sets the ceiling on what the policy can learn. This guide covers the decisions that matter, from the operator's interface to the file your training job loads.

Choose the interface for the task

VR headset and hand controllers

The headset supplies the operator's head pose and the controllers supply wrist poses, triggers set grip and joysticks drive the base. Good for humanoids and bimanual tasks where the operator needs to look around.

Leader arms

A small kinematic copy of the robot arm that the operator moves by hand, with the real arm following. Very natural for manipulation and gives clean joint-space actions.

Gamepads and 3D mice

Cheap and familiar. Fine for mobile bases, simple pick-and-place and end-effector control, less so for dexterous work.

Pick one interface per task family and keep it fixed. Mixing interfaces inside one dataset changes the action distribution in ways that are hard to see and easy to regret.

Record everything the policy will see, and what it should do

Cameras

Head and wrist cameras at minimum, at a fixed resolution and frame rate. Wrist views carry most of the contact information.

Proprioception

Joint positions and velocities, gripper state and, where available, gripper force or tactile signals.

Actions

The commands actually sent to the robot, in the same space the policy will output: joint targets or end-effector poses.

Timestamps

One clock for every stream. Misaligned streams are the most common silent bug in robot datasets.

Metadata

Task, scene, objects, robot and software version, operator ID and outcome.

Design episodes, not sessions

An episode is one attempt at one task, from a defined start state to success or failure. Write the protocol down before collection starts: the reset state, the allowed variation in object placement and lighting, the success criterion, and what the operator does when something goes wrong.

Label success and failure

Mark every episode with its outcome. Failed attempts are useful for evaluation and some training methods, but they should never be mixed in unlabeled.

Re-collect bad episodes

Falls, safety stops and operator mistakes get flagged and redone, so the clean set stays clean.

Vary on purpose

Spread object positions, distractors and lighting across the protocol rather than letting variation happen by accident.

Keep operators consistent

Train every operator on the same protocol and review a sample of each person’s episodes. Different styles of solving the same task add noise.

Have trained pilots collect it for you

How many demonstrations do you need?

Honestly, it depends on the task, the model and how much variation you need to cover. Published research systems report anywhere from tens to hundreds of demonstrations per task for single-task policies, and far more for multi-task models. Start with a pilot batch, train, evaluate on held-out setups, and let the failure modes tell you which variations to collect next.

Run collection like an operation

Data collection at scale is shift work. A few habits keep throughput up without letting quality drift.

Short shifts with breaks

Teleoperation is tiring, and tired operators produce hesitant, jerky demonstrations. Rotate tasks and schedule breaks.

A reset checklist

Most lost time is resetting the scene. A written checklist with photos of the start state makes resets fast and consistent.

Daily review

Watch a sample of yesterday’s episodes each morning and feed corrections back to operators before they become habits.

Track throughput and yield

Count episodes per hour and the share that pass QA. A falling yield is the earliest sign that the protocol or the hardware needs attention.

Version the protocol

When the protocol changes, bump its version and store it with every episode, so you can filter later.

Pick a delivery format

LeRobot

Hugging Face's dataset format: tabular frames plus encoded video, easy to share on the Hub and load straight into the LeRobot training code.

RLDS

Episodes of steps stored as TFRecords, the format used by many TensorFlow pipelines and by cross-embodiment collections in the Open X-Embodiment style.

HDF5

One file per episode with arrays per stream. Simple, fast and supported everywhere, which is why many labs still default to it.

Quality checks before training

Every stream present for every frame, with no gaps or duplicated timestamps.

Actions within the robot’s limits, with no sudden jumps from dropped packets.

Outcome labels spot-checked against video.

A replay of a random sample on the robot or in simulation reproduces the motion.

RoboPilots pilots run your robot through the task, and each session is cut into episodes, labeled by task and outcome, and delivered as LeRobot, RLDS or HDF5. Vehicle drives come back as MCAP or ROS 2 bags, tagged by scenario.

Frequently asked questions

What is teleoperation data?: Recordings of a person driving a robot through a task: camera frames, joint states, forces and the commands sent, synchronized in time. Robot learning models use them as demonstrations.
Which format should I use: LeRobot, RLDS or HDF5?: Use what your training code reads. LeRobot suits Hugging Face and PyTorch workflows, RLDS suits TensorFlow and cross-embodiment pipelines, and HDF5 is the simple, universal option.
How many demonstrations does a task need?: Research systems report tens to hundreds per task for single-task policies. Collect a pilot batch, evaluate, and collect more where the policy fails.
Should failed episodes be kept?: Keep them, but label them. They help with evaluation and some training methods, and they should never sit unlabeled among successful demonstrations.

Quote a data collection project