Continual FT from lightseek kimi-k2.6-eagle3-mla (Camelot, 10k steps): val_loss -27%, weak-position acc +45-116%

README.md

@@ -1,3 +1,61 @@
 ---
-license: mit
+license: other
+base_model: lightseekorg/kimi-k2.6-eagle3-mla
+tags:
+- text-generation
+- speculative-decoding
+- eagle3
+- kimi-k2.6
+- mla
+- camelot
 ---
+
+# kimi-k2.6-eagle3-mla (continual fine-tune)
+
+Eagle3 MTP draft model with MLA (Multi-Latent Attention) for accelerating
+inference of [Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6).
+
+This checkpoint is a **continual fine-tune** of
+[lightseekorg/kimi-k2.6-eagle3-mla](https://huggingface.co/lightseekorg/kimi-k2.6-eagle3-mla),
+further trained on an in-house instruction distribution. The architecture,
+config, and tensor layout are identical to the base model, so it is a drop-in
+replacement in the same vLLM serving path.
+
+## Training Setup
+
+- **Init**: continual FT from `lightseekorg/kimi-k2.6-eagle3-mla`.
+- **Framework**: Camelot online speculative-decoding training — concurrent
+  FSDP training + vLLM rollout with cross-node hidden-state transfer over
+  Mooncake (RDMA).
+- **Topology**: 1 datagen node (Kimi-K2.6, TP=8) -> Mooncake -> 1 trainer GPU.
+- **Data**: online-generated hidden states from a ~100k-prompt instruction set
+  (each sample consumed once; ~10k samples over the run).
+- **Steps**: 10,000 (global batch size 1), LR 2e-5 cosine, seq len 4096,
+  Eagle3 TTT steps 3.
+
+## Validation (training-time, teacher-forced)
+
+Per-position draft accuracy on a fixed held-out val split, measured during
+training (teacher-forced argmax match; this is a *training-time* metric and is
+not directly comparable to runtime accept-length). Continual FT lifts the
+weaker later positions most:
+
+| metric | init | final | rel. |
+| --- | --- | --- | --- |
+| val_loss | 7.69 | 5.62 | -27% |
+| full_acc@1 | 0.253 | 0.367 | +45% |
+| full_acc@2 | 0.085 | 0.184 | +116% |
+| cond_acc@1 | 0.412 | 0.559 | +36% |
+| cond_acc@2 | 0.336 | 0.499 | +49% |
+
+A runtime `accept_length` benchmark (vLLM 0.20, `num_speculative_tokens=3`) is
+pending and will be added once measured.
+
+## Quick Start (vLLM >= 0.20.0)
+
+```bash
+vllm serve moonshotai/Kimi-K2.6 \
+    --tensor-parallel-size 8 \
+    --speculative-config '{"model": "k-l-lambda/kimi-k2.6-eagle3-mla", "method": "eagle3", "num_speculative_tokens": 3}' \
+    --trust-remote-code
+```

config.json

@@ -0,0 +1,47 @@
+{
+  "architectures": [
+    "Eagle3DeepseekV2ForCausalLM"
+  ],
+  "model_type": "kimi_k2",
+  "hidden_size": 7168,
+  "intermediate_size": 18432,
+  "num_hidden_layers": 1,
+  "num_attention_heads": 64,
+  "num_key_value_heads": 64,
+  "q_lora_rank": 1536,
+  "kv_lora_rank": 512,
+  "qk_nope_head_dim": 128,
+  "qk_rope_head_dim": 64,
+  "v_head_dim": 128,
+  "hidden_act": "silu",
+  "rms_norm_eps": 1e-05,
+  "vocab_size": 163840,
+  "draft_vocab_size": 163840,
+  "torch_dtype": "bfloat16",
+  "rope_theta": 50000.0,
+  "rope_scaling": {
+    "beta_fast": 1.0,
+    "beta_slow": 1.0,
+    "factor": 64.0,
+    "mscale": 1.0,
+    "mscale_all_dim": 1.0,
+    "original_max_position_embeddings": 4096,
+    "type": "yarn"
+  },
+  "eagle_config": {
+    "eagle_aux_hidden_state_layer_ids": [
+      2,
+      30,
+      58
+    ],
+    "use_aux_hidden_state": true,
+    "use_input_layernorm_in_first_layer": true,
+    "use_last_layernorm": true,
+    "use_mtp_layernorm": false
+  },
+  "bos_token_id": 163584,
+  "eos_token_id": 163585,
+  "pad_token_id": 0,
+  "_torchspec_version": "0.1.0",
+  "max_position_embeddings": 262144
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc5e532f4dd84a3a3f58898a226ee8b1052f5cff4d542e02722d0cb7659c6df7
+size 6031210296