# LeRobotDataset v3.0 `LeRobotDataset v3.0` is a standardized format for robot learning data. It provides unified access to multi-modal time-series data, sensorimotor signals and multi‑camera video, as well as rich metadata for indexing, search, and visualization on the Hugging Face Hub. To see what a dataset in this format holds before reading about its layout, open one in the [LeRobot Dataset Visualizer](https://huggingface.co/spaces/lerobot/visualize_dataset?path=/lerobot/svla_so101_pickplace/episode_0): camera recordings, synchronized state and action charts, and every episode, in the browser. See [Explore a Dataset in the Browser](./using_dataset_tools#explore-a-dataset-in-the-browser). This docs will guide you to: - Understand the v3.0 design and directory layout - Record a dataset and push it to the Hub - Load datasets for training with `LeRobotDataset` - Stream datasets without downloading using `StreamingLeRobotDataset` - Apply image transforms for data augmentation during training - Migrate existing `v2.1` datasets to `v3.0` - Experiment with other `LeRobotDataset` formats and implementations like Lance ## What’s new in `v3` - **File-based storage**: Many episodes per Parquet/MP4 file (v2 used one file per episode). - **Relational metadata**: Episode boundaries and lookups are resolved through metadata, not filenames. - **Hub-native streaming**: Consume datasets directly from the Hub with `StreamingLeRobotDataset`. - **Lower file-system pressure**: Fewer, larger files ⇒ faster initialization and fewer issues at scale. - **Unified organization**: Clean directory layout with consistent path templates across data and videos. - **Dataset storage backends**: Datasets can also be stored as [LanceDB](https://lancedb.com) tables for random access and indexing directly from object storage (see [LanceDB dataset backend](#lancedb-dataset-backend)). ## Installation `LeRobotDataset v3.0` is included in `lerobot >= 0.4.0`. Follow the [Installation Guide](./installation) to install it. ## Record a dataset Run the command below to record a dataset with the SO-101 and push to the Hub: ```bash lerobot-record \ --robot.type=so101_follower \ --robot.port=/dev/tty.usbmodem585A0076841 \ --robot.id=my_awesome_follower_arm \ --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}" \ --teleop.type=so101_leader \ --teleop.port=/dev/tty.usbmodem58760431551 \ --teleop.id=my_awesome_leader_arm \ --display_data=true \ --dataset.repo_id=${HF_USER}/record-test \ --dataset.num_episodes=5 \ --dataset.single_task="Grab the black cube" \ --dataset.streaming_encoding=true \ --dataset.encoder_threads=2 # Optional: --dataset.rgb_encoder.vcodec=auto ``` See the [recording guide](./il_robots#record-a-dataset) for more details. ## Format design A core v3 principle is **decoupling storage from the user API**: data is stored efficiently (few large files), while the public API exposes intuitive episode-level access. `v3` has three pillars: 1. **Tabular data**: Low‑dimensional, high‑frequency signals (states, actions, timestamps) stored in **Apache Parquet**. Access is memory‑mapped or streamed via the `datasets` stack. 2. **Visual data**: Camera frames concatenated and encoded into **MP4**. Frames from the same episode are grouped; videos are sharded per camera for practical sizes. 3. **Metadata**: JSON/Parquet records describing schema (feature names, dtypes, shapes), frame rates, normalization stats, and **episode segmentation** (start/end offsets into shared Parquet/MP4 files). > To scale to millions of episodes, tabular rows and video frames from multiple episodes are **concatenated** into larger files. Episode‑specific views are reconstructed **via metadata**, not file boundaries. LeRobotDataset v3 diagram From episode‑based to file‑based datasets ### Directory layout (simplified) - **`meta/info.json`**: canonical schema (features, shapes/dtypes), FPS, codebase version, and **path templates** to locate data/video shards. - **`meta/stats.json`**: global feature statistics (mean/std/min/max) used for normalization; exposed as `dataset.meta.stats`. - **`meta/tasks.parquet`**: natural‑language task descriptions mapped to integer IDs for task‑conditioned policies. - **`meta/episodes/`**: per‑episode records (lengths, tasks, offsets) stored as **chunked Parquet** for scalability. - **`data/`**: frame‑by‑frame **Parquet** shards; each file typically contains **many episodes**. - **`videos/`**: **MP4** shards per camera; each file typically contains **many episodes**. ## Load a dataset for training `LeRobotDataset` returns Python dictionaries of PyTorch tensors and integrates with `torch.utils.data.DataLoader`. Here is a code example showing its use: ```python import torch from lerobot.datasets import LeRobotDataset repo_id = "yaak-ai/L2D-v3" # 1) Load from the Hub (cached locally) dataset = LeRobotDataset(repo_id) # 2) Random access by index sample = dataset[100] print(sample) # { # 'observation.state': tensor([...]), # 'action': tensor([...]), # 'observation.images.front_left': tensor([C, H, W]), # 'timestamp': tensor(1.234), # ... # } # 3) Temporal windows via delta_timestamps (seconds relative to t) delta_timestamps = { "observation.images.front_left": [-0.2, -0.1, 0.0] # 0.2s and 0.1s before current frame } dataset = LeRobotDataset(repo_id, delta_timestamps=delta_timestamps) # Accessing an index now returns a stack for the specified key(s) sample = dataset[100] print(sample["observation.images.front_left"].shape) # [T, C, H, W], where T=3 # 4) Wrap with a DataLoader for training batch_size = 16 data_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size) device = "cuda" if torch.cuda.is_available() else "cpu" for batch in data_loader: observations = batch["observation.state"].to(device) actions = batch["action"].to(device) images = batch["observation.images.front_left"].to(device) # model.forward(batch) ``` ## Stream a dataset during training Set `--dataset.streaming=true` to train without first downloading the dataset's Parquet and video files. Episodes are fetched on the fly into a bounded in-memory pool, and every selected frame is still visited exactly once per epoch: ```bash lerobot-train \ --dataset.repo_id=yaak-ai/L2D-v3 \ --dataset.streaming=true \ --policy.type=act \ --output_dir=outputs/train/act_streaming ``` The same reader is available in Python, for Hub datasets and for datasets stored in an [HF Storage Bucket](https://huggingface.co/docs/hub/storage-buckets): ```python from lerobot.datasets import StreamingLeRobotDataset dataset = StreamingLeRobotDataset("yaak-ai/L2D-v3") bucket_dataset = StreamingLeRobotDataset("my-org/my-bucket", repo_type="bucket") ``` From the CLI, select a bucket with `--dataset.repo_type=bucket` or `--dataset.root=hf://buckets/my-org/my-bucket`. See [Training Dataset Streaming](./training_dataset_streaming) for how workers and shards map to streaming concurrency, the MP4 sidecar index, memory settings, resume and troubleshooting. StreamingLeRobotDataset Stream directly from the Hub for on‑the‑fly training. ## Image transforms Image transforms are data augmentations applied to camera frames during training to improve model robustness and generalization. LeRobot supports various transforms including brightness, contrast, saturation, hue, and sharpness adjustments. ### Using transforms during dataset creation/recording Currently, transforms are applied during **training time only**, not during recording. When you create or record a dataset, the raw images are stored without transforms. This allows you to experiment with different augmentations later without re-recording data. ### Adding transforms to existing datasets (API) Use the `image_transforms` parameter when loading a dataset for training: ```python from lerobot.datasets import LeRobotDataset from lerobot.transforms import ImageTransforms, ImageTransformsConfig, ImageTransformConfig # Option 1: Use default transform configuration (disabled by default) transforms_config = ImageTransformsConfig( enable=True, # Enable transforms max_num_transforms=3, # Apply up to 3 transforms per frame random_order=False, # Apply in standard order ) transforms = ImageTransforms(transforms_config) dataset = LeRobotDataset( repo_id="your-username/your-dataset", image_transforms=transforms ) # Option 2: Create custom transform configuration custom_transforms_config = ImageTransformsConfig( enable=True, max_num_transforms=2, random_order=True, tfs={ "brightness": ImageTransformConfig( weight=1.0, type="ColorJitter", kwargs={"brightness": (0.7, 1.3)} # Adjust brightness range ), "contrast": ImageTransformConfig( weight=2.0, # Higher weight = more likely to be selected type="ColorJitter", kwargs={"contrast": (0.8, 1.2)} ), "sharpness": ImageTransformConfig( weight=0.5, # Lower weight = less likely to be selected type="SharpnessJitter", kwargs={"sharpness": (0.3, 2.0)} ), } ) dataset = LeRobotDataset( repo_id="your-username/your-dataset", image_transforms=ImageTransforms(custom_transforms_config) ) # Option 3: Use pure torchvision transforms from torchvision.transforms import v2 torchvision_transforms = v2.Compose([ v2.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1), v2.GaussianBlur(kernel_size=3, sigma=(0.1, 2.0)), ]) dataset = LeRobotDataset( repo_id="your-username/your-dataset", image_transforms=torchvision_transforms ) ``` ### Available transform types The `type` of a transform can be any class from `torchvision.transforms.v2`, for example: - **`ColorJitter`**: Adjusts brightness, contrast, saturation, and hue - **`SharpnessJitter`**: Randomly adjusts image sharpness - **`RandomAffine`**, **`RandomRotation`**: Geometric jitter - **`GaussianNoise`**, **`MotionBlur`**, **`JPEGCompression`**, **`GaussianPatchBrightness`**, **`RandomShadow`**, **`CoarseDropout`**, **`GammaCorrection`**, **`PlanckianJitter`**: Camera and lighting effects seen on real robots - **`Identity`**: No transformation (useful for testing) Any other `torchvision.transforms.v2` class can be named as a `type` too (for example `RandomResizedCrop` or `GaussianBlur`); it is applied one sample at a time. You can also pass a torchvision transform directly to the `image_transforms` parameter. Every transform is now batched, and the same batched transforms run on both backends. The per-sample classes (`SharpnessJitter`, `GaussianNoise`, `MotionBlur`, `GaussianPatchBrightness`, `RandomShadow`, `CoarseDropout`, `GammaCorrection`, `PlanckianJitter`, `RandomSubsetApply`) and `make_transform_from_config` remain importable as deprecated front ends over their `Batched*` replacements and warn on use; they will be removed in a future release. `ImageTransforms` is a plain `torch.nn.Module` taking a `(C, H, W)` or `(T, C, H, W)` tensor. Configurations naming a transform by `type` are unaffected. **Behaviour change for existing recipes.** Training reads frames as `uint8`, and the LeRobot transforms other than `ColorJitter`, `RandomAffine` and `RandomRotation` used to return non-float input unchanged. Recipes enabling `GaussianNoise`, `MotionBlur`, `GaussianPatchBrightness`, `RandomShadow`, `CoarseDropout`, `GammaCorrection` or `PlanckianJitter` were therefore training without them. They now apply on both backends, so such a recipe trains differently than it did before: it gets the augmentation it asked for. Transforms that were already applying are unchanged apart from rounding, since the batched path computes in float and rounds once at the end rather than after every transform. ### Configuration options - **`enable`**: Enable/disable transforms (default: `False`) - **`max_num_transforms`**: Maximum number of transforms applied per frame (default: `3`) - **`random_order`**: Apply transforms in random order vs. standard order (default: `False`) - **`weight`**: Sampling probability for each transform (higher = more likely, if sum of weights is not 1, they will be normalized) - **`kwargs`**: Transform-specific parameters (e.g., brightness range) - **`backend`**: Where the transforms run during training, `"dataloader"` (default) or `"gpu"`; see below - **`gpu_compile`**: On the `"gpu"` backend, `torch.compile` the batched transforms (default: `True`) - **`gpu_chunk_size`**: On the `"gpu"` backend, augment this many samples at a time to bound transient memory (default: `32`) ### Running transforms on the GPU By default the transforms run per sample in the DataLoader workers. With several cameras that is often the most expensive thing the workers do, and the GPU ends up waiting for data. Set `backend=gpu` to have the workers decode only and apply the same transforms to each batch on the policy device: ```bash lerobot-train \ --dataset.repo_id=your-username/your-dataset \ --dataset.image_transforms.enable=true \ --dataset.image_transforms.backend=gpu \ --policy.type=act \ --policy.device=cuda ``` The randomization is unchanged: each sample draws its own transforms and parameters, and the frames of one camera's history share them. Augmentation is applied to training batches only, before the policy preprocessor; eval batches, rollouts and saved checkpoints are untouched. Things to know: - Requires an accelerator; `--policy.device=cpu` is refused. A dataset built by hand with its own `image_transforms` callable is refused too, since those run in the workers. - Supports every transform type the dataloader backend does. Types without a batched implementation (`JPEGCompression`, and torchvision classes other than `ColorJitter`, `RandomAffine`, `RandomRotation` and `Identity`) are applied one sample at a time inside the batch, so they do not get the speed-up; `JPEGCompression` also round-trips through the CPU. A transform that changes the frame size is refused on a batch of more than one sample. - Reproducible from `--seed`, with each process seeded differently. Augmentation does not consume the policy's random stream, and `gpu_compile` and `gpu_chunk_size` do not change what is applied. - Both backends run the same batched transforms in float and round once at the end, so a sample augmented on one equals the same sample augmented on the other given the same parameters. - `gpu_compile` costs 15-30 s once per image shape. Peak memory is about two float copies of the batch images plus a few copies of one chunk; lower `gpu_chunk_size` if memory is tight. `random_order=true` multiplies the transform time by `max_num_transforms`. ## Migrate `v2.1` → `v3.0` A converter aggregates per‑episode files into larger shards and writes episode offsets/metadata. Convert your dataset using the instructions below. ```bash # Convert an existing v2.1 dataset hosted on the Hub: python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id= ``` **What it does** - Aggregates parquet files: `episode_000000.parquet`, `episode_000001.parquet`, … → **`file-000.parquet`**, … - Aggregates mp4 files: `episode_000000.mp4`, `episode_000001.mp4`, … → **`file-000.mp4`**, … - Updates `meta/episodes/*` (chunked Parquet) with per‑episode lengths, tasks, and byte/frame offsets. ## Common Issues ### Always call `finalize()` before pushing When creating or recording datasets, you **must** call `dataset.finalize()` to properly close parquet writers. See the [PR #1903](https://github.com/huggingface/lerobot/pull/1903) for more details. ```python from lerobot.datasets import LeRobotDataset # Create dataset and record episodes dataset = LeRobotDataset.create(...) for episode in range(num_episodes): # Record frames for frame in episode_data: dataset.add_frame(frame) dataset.save_episode() # Call finalize() when done recording and before push_to_hub() dataset.finalize() # Closes parquet writers, writes metadata footers dataset.push_to_hub() ``` **Why is this necessary?** Dataset v3.0 uses incremental parquet writing with buffered metadata for efficiency. The `finalize()` method: - Flushes any buffered episode metadata to disk - Closes parquet writers to write footer metadata, otherwise the parquet files will be corrupt - Ensures the dataset is valid for loading Without calling `finalize()`, your parquet files will be incomplete and the dataset won't load properly. ## Other formats and implementations ### LanceDB dataset backend A `v3` dataset can also be stored as [LanceDB](https://lancedb.com) tables. LanceDB is a multimodal data lakehouse built on the Lance format for AI access patterns: fast random access to any row and its video bytes, and scalar, vector and full-text indexes on the same tables, so EDA, mining, curation and training run directly from object storage. The layout keeps `meta/` unchanged and adds a `frames` table for the tabular data and a `videos` table holding the MP4s as blobs. `"storage_format": "lance"` in `meta/info.json` selects the reader; the `LeRobotDataset` API, the items and the training pipeline are unchanged. - **Train straight from the Hub or HF Storage Buckets with the GPUs fed.** Random frames are read in place, only the bytes each batch needs, so every batch is a global shuffle and nothing is downloaded first. - **Convert at about the cost of a file copy.** Videos are not re-encoded, so items are bit-identical to the source. DROID converts in 34 minutes. - **Zero-copy feature engineering.** Add columns such as scores, embeddings or corrected labels to the tables the trainer reads, without rewriting the videos or copying the dataset, then filter and search on them with the indexes. The reader ignores columns it does not know. For how this works, benchmarks and examples, see the [LanceDB LeRobot guide](https://docs.lancedb.com/training/lerobot). Camera features must be video-backed. Datasets that store camera frames as images are not supported by the reader and need to be re-encoded as video before conversion. Reading Lance datasets needs the `lancedb` extra (`pip install "lerobot[lancedb]"`). The format is read-only: record and edit datasets in the default format, then convert them with the [`lerobot-lancedb`](https://pypi.org/project/lerobot-lancedb/) package: ```bash pip install "lerobot-lancedb" lerobot-lance-convert --repo-id lerobot/pusht --out ./pusht-lance # or --root ``` Loading uses the same class: ```python from lerobot.datasets import LeRobotDataset # Hub dataset repo dataset = LeRobotDataset("lance-format/pusht-lance") # HF Storage Bucket, read in place dataset = LeRobotDataset("my-org/pusht-lance", repo_type="bucket") # the same bucket as a root URI (other object store URIs work the same way) dataset = LeRobotDataset("lerobot/pusht", root="hf://buckets/my-org/pusht-lance") ``` The HF Storage Bucket options work in `lerobot-train`: ```bash lerobot-train \ --dataset.repo_id=my-org/pusht-lance \ --dataset.repo_type=bucket \ --policy.type=act ``` Visualization works with both viewers. `lerobot-dataset-viz --repo-id lance-format/pusht-lance --episode-index 0` opens an episode in Rerun as usual, and `--display-mode foxglove` serves it to [Foxglove](https://foxglove.dev) over WebSocket. Either way only the frames on screen are fetched, with no MCAP export or download.