Hugging Face's logo

lerobot
/
Cosmos3-Nano

Robotics
LeRobot
Safetensors
cosmos3
pretrained
Model card Files
xet
 Community
HaomingSong commited on
Commit
f56938d
·
verified ·
1 Parent(s): a202fb6

Upload cosmos3-nano base checkpoint in LeRobot format

Browse files

Converted from the NVIDIA release using LeRobot b8f060837. Includes configuration, processors, tokenizer, model card and license notices.

Files changed (11) hide show
  1. .gitattributes +1 -0
  2. LICENSE +49 -0
  3. NOTICE +26 -0
  4. README.md +66 -0
  5. config.json +314 -0
  6. model.safetensors +3 -0
  7. policy_postprocessor.json +24 -0
  8. policy_preprocessor.json +85 -0
  9. tokenizer/chat_template.jinja +120 -0
  10. tokenizer/tokenizer.json +3 -0
  11. tokenizer/tokenizer_config.json +29 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ OpenMDW License Agreement, version 1.1 (OpenMDW-1.1)
2
+
3
+ By exercising rights granted to you under this agreement, you accept and agree
4
+ to its terms.
5
+
6
+ As used in this agreement, "Model Materials" means the materials provided to
7
+ you under this agreement, consisting of: (1) one or more machine learning
8
+ models (including architecture and parameters); and (2) all related artifacts
9
+ (including associated data, documentation and software) that are provided to
10
+ you hereunder.
11
+
12
+ Subject to your compliance with this agreement, permission is hereby granted,
13
+ free of charge, to deal in the Model Materials without restriction, including
14
+ under all copyright, patent, database, and trade secret rights included or
15
+ embodied therein.
16
+
17
+ If you distribute any portion of the Model Materials, you shall retain in your
18
+ distribution (1) a copy of this agreement, and (2) all copyright notices and
19
+ other notices of origin included in the Model Materials that are applicable to
20
+ your distribution.
21
+
22
+ If you file, maintain, or voluntarily participate in a lawsuit against any
23
+ person or entity asserting that the Model Materials directly or indirectly
24
+ infringe any patent or copyright, then all rights and grants made to you
25
+ hereunder are terminated, unless that lawsuit was in response to a
26
+ corresponding lawsuit first brought against you.
27
+
28
+ This agreement does not impose any restrictions or obligations with respect to
29
+ any use, modification, or sharing of any outputs generated by using the Model
30
+ Materials.
31
+
32
+ THE MODEL MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
33
+ OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
34
+ FITNESS FOR A PARTICULAR PURPOSE, TITLE, NONINFRINGEMENT, ACCURACY, OR THE
35
+ ABSENCE OF LATENT OR OTHER DEFECTS OR ERRORS, WHETHER OR NOT DISCOVERABLE, ALL
36
+ TO THE GREATEST EXTENT PERMISSIBLE UNDER APPLICABLE LAW.
37
+
38
+ YOU ARE SOLELY RESPONSIBLE FOR (1) CLEARING RIGHTS OF OTHER PERSONS THAT MAY
39
+ APPLY TO THE MODEL MATERIALS OR ANY USE THEREOF, INCLUDING WITHOUT LIMITATION
40
+ ANY PERSON'S COPYRIGHTS OR OTHER RIGHTS INCLUDED OR EMBODIED IN THE MODEL
41
+ MATERIALS; (2) OBTAINING ANY NECESSARY CONSENTS, PERMISSIONS OR OTHER RIGHTS
42
+ REQUIRED FOR ANY USE OF THE MODEL MATERIALS; OR (3) PERFORMING ANY DUE
43
+ DILIGENCE OR UNDERTAKING ANY OTHER INVESTIGATIONS INTO THE MODEL MATERIALS OR
44
+ ANYTHING INCORPORATED OR EMBODIED THEREIN.
45
+
46
+ IN NO EVENT SHALL THE PROVIDERS OF THE MODEL MATERIALS BE LIABLE FOR ANY CLAIM,
47
+ DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
48
+ OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE MODEL MATERIALS, THE
49
+ USE THEREOF OR OTHER DEALINGS THEREIN.
NOTICE ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Cosmos3-Nano base checkpoint in LeRobot format
2
+
3
+ Original model developer: NVIDIA.
4
+ Original model family: NVIDIA Cosmos(TM).
5
+ Original model repository: nvidia/Cosmos3-Nano
6
+ Exact original revision: e59a53c25979a090fa8706c9acc0c254a6e89b92
7
+ Original release: https://huggingface.co/nvidia/Cosmos3-Nano/tree/e59a53c25979a090fa8706c9acc0c254a6e89b92
8
+ Original model card: https://huggingface.co/nvidia/Cosmos3-Nano/blob/e59a53c25979a090fa8706c9acc0c254a6e89b92/README.md
9
+ Model license: OpenMDW License Agreement, version 1.1 (OpenMDW-1.1).
10
+
11
+ This distribution converts the NVIDIA release into the LeRobot robot-policy
12
+ checkpoint layout. It retains the modules required by that policy interface,
13
+ adds LeRobot configuration and serialized processors, and does not add
14
+ task-specific fine-tuning. The full native Cosmos inference API is not provided.
15
+
16
+ Conversion date: 2026-10-28 UTC.
17
+ LeRobot source revision: b8f060837ced0a4ce4b136ea65b4d56bc7894bed.
18
+ Conversion script SHA256:
19
+ a8660731d95e98c742ca3f813a493b8be12271e404364ee64d1d3bfc9468aa35
20
+
21
+ The OpenMDW 1.1 text is included without modification as LICENSE. Its source is
22
+ the official NVIDIA Cosmos repository at:
23
+ https://github.com/NVIDIA/cosmos/blob/c3e4354c96cc8c9c311babc93b51edc6fea43fb9/LICENSE
24
+
25
+ Runtime software is installed separately. The licenses of those dependencies
26
+ continue to apply to their own materials.
README.md ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: lerobot
3
+ pipeline_tag: robotics
4
+ license: other
5
+ license_name: openmdw1.1-license
6
+ license_link: https://openmdw.ai/license/1-1/
7
+ base_model: nvidia/Cosmos3-Nano
8
+ tags:
9
+ - lerobot
10
+ - cosmos3
11
+ - robotics
12
+ - pretrained
13
+ ---
14
+
15
+ # Cosmos3-Nano base for LeRobot
16
+
17
+ This repository contains the LeRobot policy conversion of the NVIDIA Cosmos3-Nano base checkpoint. It is a starting point for robot policy post-training. The conversion adds no task-specific fine-tuning.
18
+
19
+ The policy jointly predicts robot action chunks and future video latents from language and current camera observations, with optional current-state conditioning. It contains the model components needed by the LeRobot policy interface.
20
+
21
+ | Item | Value |
22
+ |---|---|
23
+ | Original model developer | NVIDIA |
24
+ | Original checkpoint | [nvidia/Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano/tree/e59a53c25979a090fa8706c9acc0c254a6e89b92) |
25
+ | Exact source revision | `e59a53c25979a090fa8706c9acc0c254a6e89b92` |
26
+ | LeRobot compatibility baseline | `b8f060837ced0a4ce4b136ea65b4d56bc7894bed` |
27
+ | Policy type | `cosmos3_nano` |
28
+ | Backbone | Qwen3-VL MoT |
29
+ | Decoder layers / hidden width | 36 / 4096 |
30
+ | Stored scalar weights, including frozen components and VAE | 16,454,213,580 |
31
+ | Stored tensors | 1356, comprising 1352 BF16 and 4 FP32 |
32
+
33
+ The stored scalar count is not a count of trainable or active parameters. The package includes a Wan2.2 VAE, tokenizer files, and serialized LeRobot processors.
34
+
35
+ ## Loading and post-training
36
+
37
+ Use a LeRobot version containing the Cosmos3 policy types. Conversion and CPU loading checks used the exact commit above.
38
+
39
+ The saved observation and action features are empty. Frame rate and camera size are unresolved. A target dataset must provide these values before constructing the policy. Read the configuration with:
40
+
41
+ ```python
42
+ from lerobot.configs.policies import PreTrainedConfig
43
+
44
+ config = PreTrainedConfig.from_pretrained("lerobot/cosmos3-nano")
45
+ ```
46
+
47
+ The training entry points select different configurations:
48
+
49
+ - `--policy.path=lerobot/cosmos3-nano` preserves the saved DROID post-training recipe, including domain 8, 32-step chunks, state conditioning, identity normalization, UniPC with 4 inference steps, and guidance 3. State and action must already use the same layout and width.
50
+ - `--policy.type=cosmos3_nano --policy.pretrained_path=lerobot/cosmos3-nano` uses these weights with the size class's default LIBERO recipe, including domain 5, 16-step chunks, no state conditioning, action quantile normalization, UniPC with 30 steps, and guidance 1. The dataset must match that recipe's camera and action layout.
51
+
52
+ `load_action_modules` defaults to `true`, preserving the released action modules. For native-style post-training, `--policy.load_action_modules=false` reinitializes the action input projection, action output projection, and action modality embedding. The setting returns to `true` after loading so subsequent reloads preserve trained action modules.
53
+
54
+ The saved `dtype=float32` supports FP32 master weights during training. Most stored weights retain upstream BF16 precision. The four time-embedder tensors are FP32. For inference set `--policy.dtype=bfloat16`. For evaluation also set `--tf32=false`.
55
+
56
+ ## Scope and evaluation
57
+
58
+ The LeRobot implementation supports joint video-and-action `wam` mode, a single observation step, and the UniPC sampler. Actual action width comes from the target dataset. This package does not expose the upstream reasoning, audio, or standalone video-generation APIs.
59
+
60
+ CPU checks verified conversion and strict loading with resolved dataset features. No LIBERO closed-loop success rate is reported for this base conversion. Results for separately fine-tuned LIBERO checkpoints or NVIDIA Policy-DROID models do not measure this checkpoint.
61
+
62
+ ## License and provenance
63
+
64
+ The NVIDIA model materials use [OpenMDW 1.1](LICENSE). Retain the license and applicable copyright and origin notices when redistributing them. [NOTICE](NOTICE) records the original model and conversion attribution. Separately installed runtime software retains its own licenses.
65
+
66
+ Consult the original [NVIDIA model card](https://huggingface.co/nvidia/Cosmos3-Nano/blob/e59a53c25979a090fa8706c9acc0c254a6e89b92/README.md) and its [safety](https://huggingface.co/nvidia/Cosmos3-Nano/blob/e59a53c25979a090fa8706c9acc0c254a6e89b92/SAFETY.md), [bias](https://huggingface.co/nvidia/Cosmos3-Nano/blob/e59a53c25979a090fa8706c9acc0c254a6e89b92/BIAS.md), [privacy](https://huggingface.co/nvidia/Cosmos3-Nano/blob/e59a53c25979a090fa8706c9acc0c254a6e89b92/PRIVACY.md), and [explainability](https://huggingface.co/nvidia/Cosmos3-Nano/blob/e59a53c25979a090fa8706c9acc0c254a6e89b92/EXPLAINABILITY.md) notes for upstream limitations.
config.json ADDED
@@ -0,0 +1,314 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "type": "cosmos3_nano",
3
+ "n_obs_steps": 1,
4
+ "input_features": {},
5
+ "output_features": {},
6
+ "device": "cuda",
7
+ "dtype": "float32",
8
+ "use_amp": false,
9
+ "use_peft": false,
10
+ "push_to_hub": false,
11
+ "repo_id": null,
12
+ "private": null,
13
+ "tags": null,
14
+ "license": null,
15
+ "pretrained_path": null,
16
+ "pretrained_revision": null,
17
+ "chunk_size": 32,
18
+ "n_action_steps": 32,
19
+ "max_action_dim": 64,
20
+ "normalization_mapping": {
21
+ "VISUAL": "IDENTITY",
22
+ "STATE": "IDENTITY",
23
+ "ACTION": "IDENTITY"
24
+ },
25
+ "mot_backbone": {
26
+ "transformers_version": "5.5.4",
27
+ "architectures": null,
28
+ "output_hidden_states": false,
29
+ "return_dict": true,
30
+ "dtype": null,
31
+ "chunk_size_feed_forward": 0,
32
+ "is_encoder_decoder": false,
33
+ "id2label": {
34
+ "0": "LABEL_0",
35
+ "1": "LABEL_1"
36
+ },
37
+ "label2id": {
38
+ "LABEL_0": 0,
39
+ "LABEL_1": 1
40
+ },
41
+ "problem_type": null,
42
+ "text_config": {
43
+ "architectures": null,
44
+ "output_hidden_states": false,
45
+ "return_dict": true,
46
+ "dtype": null,
47
+ "chunk_size_feed_forward": 0,
48
+ "is_encoder_decoder": false,
49
+ "id2label": {
50
+ "0": "LABEL_0",
51
+ "1": "LABEL_1"
52
+ },
53
+ "label2id": {
54
+ "LABEL_0": 0,
55
+ "LABEL_1": 1
56
+ },
57
+ "problem_type": null,
58
+ "vocab_size": 151936,
59
+ "hidden_size": 4096,
60
+ "intermediate_size": 12288,
61
+ "num_hidden_layers": 36,
62
+ "num_attention_heads": 32,
63
+ "num_key_value_heads": 8,
64
+ "head_dim": 128,
65
+ "hidden_act": "silu",
66
+ "max_position_embeddings": 262144,
67
+ "initializer_range": 0.02,
68
+ "rms_norm_eps": 1e-06,
69
+ "use_cache": true,
70
+ "rope_parameters": {
71
+ "mrope_interleaved": true,
72
+ "mrope_section": [
73
+ 24,
74
+ 20,
75
+ 20
76
+ ],
77
+ "rope_type": "default",
78
+ "rope_theta": 5000000
79
+ },
80
+ "attention_bias": false,
81
+ "attention_dropout": 0.0,
82
+ "pad_token_id": null,
83
+ "joint_attn_implementation": "two_way",
84
+ "_name_or_path": "",
85
+ "bos_token_id": 151643,
86
+ "eos_token_id": 151645,
87
+ "model_type": "qwen3_vl_text",
88
+ "output_attentions": false
89
+ },
90
+ "vision_config": {
91
+ "architectures": null,
92
+ "output_hidden_states": false,
93
+ "return_dict": true,
94
+ "dtype": null,
95
+ "chunk_size_feed_forward": 0,
96
+ "is_encoder_decoder": false,
97
+ "id2label": {
98
+ "0": "LABEL_0",
99
+ "1": "LABEL_1"
100
+ },
101
+ "label2id": {
102
+ "LABEL_0": 0,
103
+ "LABEL_1": 1
104
+ },
105
+ "problem_type": null,
106
+ "depth": 27,
107
+ "hidden_size": 1152,
108
+ "hidden_act": "gelu_pytorch_tanh",
109
+ "intermediate_size": 4304,
110
+ "num_heads": 16,
111
+ "in_channels": 3,
112
+ "patch_size": 16,
113
+ "spatial_merge_size": 2,
114
+ "temporal_patch_size": 2,
115
+ "out_hidden_size": 4096,
116
+ "num_position_embeddings": 2304,
117
+ "deepstack_visual_indexes": [
118
+ 8,
119
+ 16,
120
+ 24
121
+ ],
122
+ "initializer_range": 0.02,
123
+ "_name_or_path": "",
124
+ "model_type": "qwen3_vl",
125
+ "output_attentions": false
126
+ },
127
+ "image_token_id": 151655,
128
+ "video_token_id": 151656,
129
+ "vision_start_token_id": 151652,
130
+ "vision_end_token_id": 151653,
131
+ "tie_word_embeddings": false,
132
+ "_name_or_path": "",
133
+ "model_type": "qwen3_vl",
134
+ "output_attentions": false
135
+ },
136
+ "tokenizer_name": "nvidia/Cosmos3-Nano",
137
+ "vae": {
138
+ "base_dim": 160,
139
+ "decoder_base_dim": 256,
140
+ "z_dim": 48,
141
+ "dim_mult": [
142
+ 1,
143
+ 2,
144
+ 4,
145
+ 4
146
+ ],
147
+ "num_res_blocks": 2,
148
+ "attn_scales": [],
149
+ "temperal_downsample": [
150
+ false,
151
+ true,
152
+ true
153
+ ],
154
+ "dropout": 0.0,
155
+ "latents_mean": [
156
+ -0.2289,
157
+ -0.0052,
158
+ -0.1323,
159
+ -0.2339,
160
+ -0.2799,
161
+ 0.0174,
162
+ 0.1838,
163
+ 0.1557,
164
+ -0.1382,
165
+ 0.0542,
166
+ 0.2813,
167
+ 0.0891,
168
+ 0.157,
169
+ -0.0098,
170
+ 0.0375,
171
+ -0.1825,
172
+ -0.2246,
173
+ -0.1207,
174
+ -0.0698,
175
+ 0.5109,
176
+ 0.2665,
177
+ -0.2108,
178
+ -0.2158,
179
+ 0.2502,
180
+ -0.2055,
181
+ -0.0322,
182
+ 0.1109,
183
+ 0.1567,
184
+ -0.0729,
185
+ 0.0899,
186
+ -0.2799,
187
+ -0.123,
188
+ -0.0313,
189
+ -0.1649,
190
+ 0.0117,
191
+ 0.0723,
192
+ -0.2839,
193
+ -0.2083,
194
+ -0.052,
195
+ 0.3748,
196
+ 0.0152,
197
+ 0.1957,
198
+ 0.1433,
199
+ -0.2944,
200
+ 0.3573,
201
+ -0.0548,
202
+ -0.1681,
203
+ -0.0667
204
+ ],
205
+ "latents_std": [
206
+ 0.4765,
207
+ 1.0364,
208
+ 0.4514,
209
+ 1.1677,
210
+ 0.5313,
211
+ 0.499,
212
+ 0.4818,
213
+ 0.5013,
214
+ 0.8158,
215
+ 1.0344,
216
+ 0.5894,
217
+ 1.0901,
218
+ 0.6885,
219
+ 0.6165,
220
+ 0.8454,
221
+ 0.4978,
222
+ 0.5759,
223
+ 0.3523,
224
+ 0.7135,
225
+ 0.6804,
226
+ 0.5833,
227
+ 1.4146,
228
+ 0.8986,
229
+ 0.5659,
230
+ 0.7069,
231
+ 0.5338,
232
+ 0.4889,
233
+ 0.4917,
234
+ 0.4069,
235
+ 0.4999,
236
+ 0.6866,
237
+ 0.4093,
238
+ 0.5709,
239
+ 0.6065,
240
+ 0.6415,
241
+ 0.4944,
242
+ 0.5726,
243
+ 1.2042,
244
+ 0.5458,
245
+ 1.6887,
246
+ 0.3971,
247
+ 1.06,
248
+ 0.3943,
249
+ 0.5537,
250
+ 0.5444,
251
+ 0.4089,
252
+ 0.7468,
253
+ 0.7744
254
+ ],
255
+ "is_residual": true,
256
+ "in_channels": 12,
257
+ "out_channels": 12,
258
+ "patch_size": 2,
259
+ "scale_factor_temporal": 4,
260
+ "scale_factor_spatial": 16
261
+ },
262
+ "latent_patch_size": 2,
263
+ "timestep_scale": 0.001,
264
+ "timestep_embedding_dim": 256,
265
+ "num_embodiment_domains": 32,
266
+ "base_fps": 24.0,
267
+ "temporal_modality_margin": 15000,
268
+ "video_temporal_causal": false,
269
+ "train_gen_stream_only": true,
270
+ "freeze_vision_encoder": true,
271
+ "load_action_modules": true,
272
+ "gradient_checkpointing": false,
273
+ "compile_model": false,
274
+ "compile_mode": "default",
275
+ "attn_implementation": "sdpa",
276
+ "num_train_timesteps": 1000,
277
+ "video_time_distribution": "waver",
278
+ "training_shift": {
279
+ "256": 3.0,
280
+ "480": 5.0,
281
+ "720": 10.0
282
+ },
283
+ "independent_action_schedule": false,
284
+ "action_time_distribution": "logitnormal",
285
+ "action_training_shift": null,
286
+ "video_loss_weight": 10.0,
287
+ "action_loss_weight": 10.0,
288
+ "num_inference_steps": 4,
289
+ "guidance_scale": 3.0,
290
+ "guidance_interval": null,
291
+ "sampling_shift": 5.0,
292
+ "sampler": "unipc",
293
+ "save_predicted_video": false,
294
+ "mode": "wam",
295
+ "use_state": true,
296
+ "domain_id": 8,
297
+ "fps": null,
298
+ "view_size": null,
299
+ "view_description": "The top row is from the wrist-mounted camera. The bottom row contains two horizontally concatenated third-person perspective views of the scene from opposite sides, with the robot visible.",
300
+ "idle_state": true,
301
+ "action_feature_names": null,
302
+ "cfg_dropout_rate": 0.1,
303
+ "optimizer_lr": 8e-05,
304
+ "optimizer_betas": [
305
+ 0.9,
306
+ 0.99
307
+ ],
308
+ "optimizer_eps": 1e-08,
309
+ "optimizer_weight_decay": 0.05,
310
+ "optimizer_grad_clip_norm": 1.0,
311
+ "optimizer_action_lr_multiplier": 5.0,
312
+ "scheduler_warmup_steps": 0,
313
+ "scheduler_decay_steps": 100000
314
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:44e9286b0ebcb15bf852606abbafd41dcef3576b35924f9efe8b7f6714f1ea71
3
+ size 32944302256
policy_postprocessor.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "policy_postprocessor",
3
+ "steps": [
4
+ {
5
+ "registry_name": "unnormalizer_processor",
6
+ "config": {
7
+ "eps": 1e-08,
8
+ "features": {},
9
+ "norm_map": {
10
+ "VISUAL": "IDENTITY",
11
+ "STATE": "IDENTITY",
12
+ "ACTION": "IDENTITY"
13
+ }
14
+ }
15
+ },
16
+ {
17
+ "registry_name": "device_processor",
18
+ "config": {
19
+ "device": "cuda",
20
+ "float_dtype": null
21
+ }
22
+ }
23
+ ]
24
+ }
policy_preprocessor.json ADDED
@@ -0,0 +1,85 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "policy_preprocessor",
3
+ "steps": [
4
+ {
5
+ "registry_name": "rename_observations_processor",
6
+ "config": {
7
+ "rename_map": {}
8
+ }
9
+ },
10
+ {
11
+ "registry_name": "to_batch_processor",
12
+ "config": {}
13
+ },
14
+ {
15
+ "registry_name": "device_processor",
16
+ "config": {
17
+ "device": "cuda",
18
+ "float_dtype": null
19
+ }
20
+ },
21
+ {
22
+ "registry_name": "cosmos3_idle_frame_processor",
23
+ "config": {
24
+ "action_feature_names": null,
25
+ "fps": null,
26
+ "use_state": true,
27
+ "idle_state": true,
28
+ "translation_threshold": 0.005,
29
+ "rotation_threshold": 0.026179938779914945,
30
+ "gripper_threshold": 0.01,
31
+ "joint_threshold": 0.005,
32
+ "min_streak": 3
33
+ }
34
+ },
35
+ {
36
+ "registry_name": "normalizer_processor",
37
+ "config": {
38
+ "eps": 1e-08,
39
+ "features": {},
40
+ "norm_map": {
41
+ "VISUAL": "IDENTITY",
42
+ "STATE": "IDENTITY",
43
+ "ACTION": "IDENTITY"
44
+ }
45
+ }
46
+ },
47
+ {
48
+ "registry_name": "cosmos3_video_canvas_processor",
49
+ "config": {
50
+ "camera_keys": [],
51
+ "view_size": null,
52
+ "fps": null,
53
+ "chunk_size": 32,
54
+ "temporal_compression": 4,
55
+ "spatial_compression": 16
56
+ }
57
+ },
58
+ {
59
+ "registry_name": "cosmos3_prompt_processor",
60
+ "config": {
61
+ "view_description": "The top row is from the wrist-mounted camera. The bottom row contains two horizontally concatenated third-person perspective views of the scene from opposite sides, with the robot visible.",
62
+ "fps": null,
63
+ "chunk_size": 32,
64
+ "use_state": true
65
+ }
66
+ },
67
+ {
68
+ "registry_name": "cosmos3_tokenizer_processor",
69
+ "config": {
70
+ "tokenizer_name": "tokenizer",
71
+ "cfg_dropout_rate": 0.1
72
+ },
73
+ "artifacts": {
74
+ "tokenizer_name": "tokenizer"
75
+ }
76
+ },
77
+ {
78
+ "registry_name": "device_processor",
79
+ "config": {
80
+ "device": "cuda",
81
+ "float_dtype": null
82
+ }
83
+ }
84
+ ]
85
+ }
tokenizer/chat_template.jinja ADDED
@@ -0,0 +1,120 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- if tools %}
2
+ {{- '<|im_start|>system\n' }}
3
+ {%- if messages[0].role == 'system' %}
4
+ {%- if messages[0].content is string %}
5
+ {{- messages[0].content }}
6
+ {%- else %}
7
+ {%- for content in messages[0].content %}
8
+ {%- if 'text' in content %}
9
+ {{- content.text }}
10
+ {%- endif %}
11
+ {%- endfor %}
12
+ {%- endif %}
13
+ {{- '\n\n' }}
14
+ {%- endif %}
15
+ {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
16
+ {%- for tool in tools %}
17
+ {{- "\n" }}
18
+ {{- tool | tojson }}
19
+ {%- endfor %}
20
+ {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
21
+ {%- else %}
22
+ {%- if messages[0].role == 'system' %}
23
+ {{- '<|im_start|>system\n' }}
24
+ {%- if messages[0].content is string %}
25
+ {{- messages[0].content }}
26
+ {%- else %}
27
+ {%- for content in messages[0].content %}
28
+ {%- if 'text' in content %}
29
+ {{- content.text }}
30
+ {%- endif %}
31
+ {%- endfor %}
32
+ {%- endif %}
33
+ {{- '<|im_end|>\n' }}
34
+ {%- endif %}
35
+ {%- endif %}
36
+ {%- set image_count = namespace(value=0) %}
37
+ {%- set video_count = namespace(value=0) %}
38
+ {%- for message in messages %}
39
+ {%- if message.role == "user" %}
40
+ {{- '<|im_start|>' + message.role + '\n' }}
41
+ {%- if message.content is string %}
42
+ {{- message.content }}
43
+ {%- else %}
44
+ {%- for content in message.content %}
45
+ {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
46
+ {%- set image_count.value = image_count.value + 1 %}
47
+ {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
48
+ <|vision_start|><|image_pad|><|vision_end|>
49
+ {%- elif content.type == 'video' or 'video' in content %}
50
+ {%- set video_count.value = video_count.value + 1 %}
51
+ {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
52
+ <|vision_start|><|video_pad|><|vision_end|>
53
+ {%- elif 'text' in content %}
54
+ {{- content.text }}
55
+ {%- endif %}
56
+ {%- endfor %}
57
+ {%- endif %}
58
+ {{- '<|im_end|>\n' }}
59
+ {%- elif message.role == "assistant" %}
60
+ {{- '<|im_start|>' + message.role + '\n' }}
61
+ {%- if message.content is string %}
62
+ {{- message.content }}
63
+ {%- else %}
64
+ {%- for content_item in message.content %}
65
+ {%- if 'text' in content_item %}
66
+ {{- content_item.text }}
67
+ {%- endif %}
68
+ {%- endfor %}
69
+ {%- endif %}
70
+ {%- if message.tool_calls %}
71
+ {%- for tool_call in message.tool_calls %}
72
+ {%- if (loop.first and message.content) or (not loop.first) %}
73
+ {{- '\n' }}
74
+ {%- endif %}
75
+ {%- if tool_call.function %}
76
+ {%- set tool_call = tool_call.function %}
77
+ {%- endif %}
78
+ {{- '<tool_call>\n{"name": "' }}
79
+ {{- tool_call.name }}
80
+ {{- '", "arguments": ' }}
81
+ {%- if tool_call.arguments is string %}
82
+ {{- tool_call.arguments }}
83
+ {%- else %}
84
+ {{- tool_call.arguments | tojson }}
85
+ {%- endif %}
86
+ {{- '}\n</tool_call>' }}
87
+ {%- endfor %}
88
+ {%- endif %}
89
+ {{- '<|im_end|>\n' }}
90
+ {%- elif message.role == "tool" %}
91
+ {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
92
+ {{- '<|im_start|>user' }}
93
+ {%- endif %}
94
+ {{- '\n<tool_response>\n' }}
95
+ {%- if message.content is string %}
96
+ {{- message.content }}
97
+ {%- else %}
98
+ {%- for content in message.content %}
99
+ {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
100
+ {%- set image_count.value = image_count.value + 1 %}
101
+ {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
102
+ <|vision_start|><|image_pad|><|vision_end|>
103
+ {%- elif content.type == 'video' or 'video' in content %}
104
+ {%- set video_count.value = video_count.value + 1 %}
105
+ {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
106
+ <|vision_start|><|video_pad|><|vision_end|>
107
+ {%- elif 'text' in content %}
108
+ {{- content.text }}
109
+ {%- endif %}
110
+ {%- endfor %}
111
+ {%- endif %}
112
+ {{- '\n</tool_response>' }}
113
+ {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
114
+ {{- '<|im_end|>\n' }}
115
+ {%- endif %}
116
+ {%- endif %}
117
+ {%- endfor %}
118
+ {%- if add_generation_prompt %}
119
+ {{- '<|im_start|>assistant\n' }}
120
+ {%- endif %}
tokenizer/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
3
+ size 11422650
tokenizer/tokenizer_config.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": null,
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": [
9
+ "<|im_start|>",
10
+ "<|im_end|>",
11
+ "<|object_ref_start|>",
12
+ "<|object_ref_end|>",
13
+ "<|box_start|>",
14
+ "<|box_end|>",
15
+ "<|quad_start|>",
16
+ "<|quad_end|>",
17
+ "<|vision_start|>",
18
+ "<|vision_end|>",
19
+ "<|vision_pad|>",
20
+ "<|image_pad|>",
21
+ "<|video_pad|>"
22
+ ],
23
+ "is_local": true,
24
+ "model_max_length": 262144,
25
+ "pad_token": "<|endoftext|>",
26
+ "split_special_tokens": false,
27
+ "tokenizer_class": "Qwen2Tokenizer",
28
+ "unk_token": null
29
+ }