Qwen/Qwen3-8B — lems search — 0.6 target ratio

This model was compressed using kfac_svd with lems rank search starting from Qwen/Qwen3-8B as base model. You may check out our paper, the publication, the code, and project page for details on kfac-svd and our LEMS rank search.

Update (2026-10-23). The weights were recomputed with the fixed KFAC-SVD implementation. The first release of the code (v1, used for the published paper) whitened the output side of each weight with the Cholesky factor L of G = LLᵀ instead of Lᵀ. The correct version was described by the algorithm in the paper already. Those weights, which the paper's tables report, remain available as revision v1-paper: AutoModelForCausalLM.from_pretrained("MoritzMo123/kfac-svd_lems_Qwen3-8B_0.6", revision="v1-paper", trust_remote_code=True).

Compression Details

Metric Value
Base Model Qwen/Qwen3-8B
Method kfac_svd
Search Method lems
Target Ratio 0.6
Compression Metric params
Recommended Dtype float16
Compressed Layers 158
Total Parameters 5,412,361,374

Usage

The checkpoint records its recommended dtype in config.json; no explicit torch_dtype argument should be needed with this remote-code wrapper. For standard Transformers models, torch_dtype="auto" is the portable fallback.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "MoritzMo123/kfac-svd_lems_Qwen3-8B_0.6",
    trust_remote_code=True,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("MoritzMo123/kfac-svd_lems_Qwen3-8B_0.6")

inputs = tokenizer('Hello, ', return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation Results

Dataset Perplexity
wikitext2 15.07
ptb 112.91
c4 54.39

Zero-shot accuracy (LM-Eval, %):

Task Accuracy
OpenBookQA 22.80
ARC-e 61.83
ARC-c 29.69
WinoGrande 59.19
HellaSwag 35.30
PIQA 64.80
Average 45.60

Rank Allocation

Per-layer ranks (click to expand)
Layer Rank
model.layers.0.mlp.gate_proj 1744
model.layers.0.mlp.up_proj 1752
model.layers.1.mlp.down_proj 976
model.layers.1.mlp.gate_proj 976
model.layers.1.mlp.up_proj 1624
model.layers.1.self_attn.k_proj 440
model.layers.1.self_attn.o_proj 648
model.layers.1.self_attn.q_proj 624
model.layers.10.self_attn.o_proj 1024
model.layers.10.self_attn.q_proj 968
model.layers.11.mlp.down_proj 2192
model.layers.11.mlp.gate_proj 1944
model.layers.11.mlp.up_proj 2136
model.layers.11.self_attn.o_proj 856
model.layers.11.self_attn.q_proj 760
model.layers.12.mlp.down_proj 1976
model.layers.12.mlp.gate_proj 1704
model.layers.12.mlp.up_proj 2160
model.layers.12.self_attn.o_proj 800
model.layers.12.self_attn.q_proj 648
model.layers.13.mlp.down_proj 1672
model.layers.13.mlp.gate_proj 1376
model.layers.13.mlp.up_proj 1792
model.layers.13.self_attn.o_proj 752
model.layers.13.self_attn.q_proj 632
model.layers.14.mlp.down_proj 1488
model.layers.14.mlp.gate_proj 1216
model.layers.14.mlp.up_proj 1608
model.layers.14.self_attn.k_proj 760
model.layers.14.self_attn.o_proj 1104
model.layers.14.self_attn.q_proj 648
model.layers.15.mlp.down_proj 1128
model.layers.15.mlp.gate_proj 976
model.layers.15.mlp.up_proj 1200
model.layers.15.self_attn.o_proj 936
model.layers.15.self_attn.q_proj 640
model.layers.16.mlp.down_proj 1424
model.layers.16.mlp.gate_proj 968
model.layers.16.mlp.up_proj 1264
model.layers.16.self_attn.o_proj 896
model.layers.16.self_attn.q_proj 648
model.layers.17.mlp.down_proj 1048
model.layers.17.mlp.gate_proj 968
model.layers.17.mlp.up_proj 1128
model.layers.17.self_attn.k_proj 672
model.layers.17.self_attn.o_proj 968
model.layers.17.self_attn.q_proj 632
model.layers.18.mlp.down_proj 944
model.layers.18.mlp.gate_proj 928
model.layers.18.mlp.up_proj 1120
model.layers.18.self_attn.k_proj 656
model.layers.18.self_attn.o_proj 920
model.layers.18.self_attn.q_proj 648
model.layers.19.mlp.down_proj 968
model.layers.19.mlp.gate_proj 968
model.layers.19.mlp.up_proj 936
model.layers.19.self_attn.o_proj 832
model.layers.19.self_attn.q_proj 640
model.layers.2.mlp.down_proj 952
model.layers.2.mlp.gate_proj 936
model.layers.2.mlp.up_proj 936
model.layers.2.self_attn.k_proj 560
model.layers.2.self_attn.o_proj 616
model.layers.2.self_attn.q_proj 632
model.layers.2.self_attn.v_proj 480
model.layers.20.mlp.down_proj 984
model.layers.20.mlp.gate_proj 944
model.layers.20.mlp.up_proj 968
model.layers.20.self_attn.k_proj 680
model.layers.20.self_attn.o_proj 712
model.layers.20.self_attn.q_proj 616
model.layers.21.mlp.down_proj 968
model.layers.21.mlp.gate_proj 1000
model.layers.21.mlp.up_proj 968
model.layers.21.self_attn.o_proj 920
model.layers.21.self_attn.q_proj 624
model.layers.22.mlp.down_proj 1448
model.layers.22.mlp.gate_proj 992
model.layers.22.mlp.up_proj 1064
model.layers.22.self_attn.o_proj 776
model.layers.22.self_attn.q_proj 624
model.layers.23.mlp.down_proj 1944
model.layers.23.mlp.gate_proj 1272
model.layers.23.mlp.up_proj 1088
model.layers.23.self_attn.o_proj 1184
model.layers.23.self_attn.q_proj 624
model.layers.24.mlp.down_proj 2008
model.layers.24.mlp.gate_proj 1232
model.layers.24.mlp.up_proj 1136
model.layers.24.self_attn.o_proj 1176
model.layers.24.self_attn.q_proj 624
model.layers.25.mlp.down_proj 2032
model.layers.25.mlp.gate_proj 1304
model.layers.25.mlp.up_proj 1280
model.layers.25.self_attn.o_proj 616
model.layers.25.self_attn.q_proj 616
model.layers.26.mlp.down_proj 2072
model.layers.26.mlp.gate_proj 1448
model.layers.26.mlp.up_proj 1336
model.layers.26.self_attn.o_proj 680
model.layers.26.self_attn.q_proj 624
model.layers.27.mlp.down_proj 2040
model.layers.27.mlp.gate_proj 1528
model.layers.27.mlp.up_proj 1392
model.layers.27.self_attn.o_proj 616
model.layers.27.self_attn.q_proj 616
model.layers.28.mlp.gate_proj 1784
model.layers.28.mlp.up_proj 1480
model.layers.28.self_attn.o_proj 616
model.layers.28.self_attn.q_proj 640
model.layers.29.mlp.gate_proj 1952
model.layers.29.mlp.up_proj 1712
model.layers.29.self_attn.o_proj 624
model.layers.29.self_attn.q_proj 616
model.layers.29.self_attn.v_proj 680
model.layers.3.mlp.down_proj 1640
model.layers.3.mlp.gate_proj 1576
model.layers.3.mlp.up_proj 936
model.layers.3.self_attn.k_proj 368
model.layers.3.self_attn.o_proj 616
model.layers.3.self_attn.q_proj 616
model.layers.30.mlp.gate_proj 2016
model.layers.30.mlp.up_proj 1872
model.layers.30.self_attn.o_proj 624
model.layers.30.self_attn.q_proj 616
model.layers.31.mlp.up_proj 2080
model.layers.31.self_attn.o_proj 616
model.layers.31.self_attn.q_proj 672
model.layers.32.self_attn.o_proj 616
model.layers.32.self_attn.q_proj 616
model.layers.32.self_attn.v_proj 648
model.layers.33.mlp.gate_proj 2096
model.layers.33.self_attn.o_proj 1008
model.layers.33.self_attn.q_proj 616
model.layers.33.self_attn.v_proj 608
model.layers.34.self_attn.o_proj 632
model.layers.34.self_attn.q_proj 640
model.layers.35.self_attn.k_proj 296
model.layers.35.self_attn.o_proj 616
model.layers.35.self_attn.q_proj 624
model.layers.4.mlp.gate_proj 1784
model.layers.4.mlp.up_proj 1512
model.layers.4.self_attn.k_proj 576
model.layers.4.self_attn.o_proj 760
model.layers.4.self_attn.q_proj 672
model.layers.5.mlp.gate_proj 1760
model.layers.5.self_attn.o_proj 648
model.layers.5.self_attn.q_proj 624
model.layers.6.mlp.gate_proj 1616
model.layers.6.self_attn.k_proj 504
model.layers.6.self_attn.o_proj 912
model.layers.6.self_attn.q_proj 672
model.layers.7.self_attn.o_proj 752
model.layers.7.self_attn.q_proj 656
model.layers.8.self_attn.o_proj 880
model.layers.8.self_attn.q_proj 624
model.layers.9.self_attn.o_proj 712
model.layers.9.self_attn.q_proj 632

Hydra Configuration Summary

Config Field Value
Model Qwen/Qwen3-8B
SVD Method kfac_svd
Search Method lems
Compression Target 0.6
Target Metric params
Calibration Dataset wikitext2
Calibration Samples 256
Sequence Length 2048
Seed 42
Downloads last month
230
Safetensors
Model size
5B params
Tensor type
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for MoritzMo123/kfac-svd_lems_Qwen3-8B_0.6

Finetuned
Qwen/Qwen3-8B
Finetuned
(2,352)
this model

Dataset used to train MoritzMo123/kfac-svd_lems_Qwen3-8B_0.6

Collection including MoritzMo123/kfac-svd_lems_Qwen3-8B_0.6

Paper for MoritzMo123/kfac-svd_lems_Qwen3-8B_0.6

Evaluation results