turboderp

/

Qwen3.6-27B-DFlash-exl3

Like 14

 DFlash num_draft_tokens (ndt) Benchmark — Qwen3.6-27B on exllamav3

 Model: Qwen3.6-27B EXL3 (4.15bpw) + DFlash draft model on 3090Ti 24GB
 Benchmark: 10 sequential requests, ~300s window each, streamed output

 ### Summary Table

 ┌──────────────────────────────────────────────────┬───────┬────────┬────────┐
 │ Metric                                           │ ndt=6 │ ndt=10 │ ndt=15 │
 ├──────────────────────────────────────────────────┼───────┼────────┼────────┤
 │ Decode tokens/sec (per-user avg)                 │ 85.3  │ 96.2   │ 81.1   │

 ┌─────────────┬───────────────────────────────────────────────────────────────────────────┬──────────────────────────────┐
 │ Role        │ HuggingFace                                                               │ Quantization                 │
 ├─────────────┼───────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤
 │ Main model  │ https://huggingface.co/UnstableLlama/Qwen3.6-27B-exl3-4.15bpw             │ EXL3, 4.15 bpw               │
 ├─────────────┼───────────────────────────────────────────────────────────────────────────┼──────────────────────────────┤
 │ Draft model │ https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3 (branch 4.00bpw) │ EXL3 DFlash tensors, 4.0 bpw │
 └─────────────┴───────────────────────────────────────────────────────────────────────────┴──────────────────────────────┘

i had to vibe code a few changes to tabbyAPI to get it working with everything on exllamav3 dev branch, and add config for num_draft_tokens.. but dropping it down to 10 from default of 15 helped a lot on this coding question aiperf concurrency=1 short test.

Decode speed on exllamav3 looks promising, and seems faster at least on this workload than regular MTP (not DFlash) on ik_llama.cpp (and the mainline draft PR which people are just using). But I haven't done a good benchmark of prefill to get a better full view.

Thanks!

did you bench this against MTP version ? https://huggingface.co/turboderp/Qwen3.6-27B-MTP-exl3

@willfalco

No, I haven't bench'd the exllamav3 MTP version. Things are moving fast, mainline llama.cpp's llama-server -sm tensor supposedly got pretty good for dual GPUs. On my single 3090TI i'm getting maybe 1400 tok/sec PP and 80-95 tok/sec TG (workload dependent) with ik_llama.cpp and my MTP IQ4_KS 4.75BPW.

Looking at the speed-ups mentioned that exl3 MTP you link could possibly be faster and similar/better quality so worth checking out.