jeffwan/llama-7b-hf — lems search — 0.8 target ratio

This model was compressed using kfac_svd with lems rank search starting from jeffwan/llama-7b-hf as base model. You may check out our publication and project page for details on kfac-svd and our LEMS rank search.

Note (2026-10-23). These weights were computed with the fixed KFAC-SVD implementation. The first release of the code (v1, used for the published paper) whitened the output side of each weight with the Cholesky factor L of G = LLᵀ instead of Lᵀ. The correct version was described by the algorithm in the paper already.

Compression Details

Metric Value
Base Model jeffwan/llama-7b-hf
Method kfac_svd
Search Method lems
Target Ratio 0.8
Compression Metric params
Recommended Dtype float16
Compressed Layers 114
Total Parameters 5,443,037,298

Usage

The checkpoint records its recommended dtype in config.json; no explicit torch_dtype argument should be needed with this remote-code wrapper. For standard Transformers models, torch_dtype="auto" is the portable fallback.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8",
    trust_remote_code=True,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8")

inputs = tokenizer('Hello, ', return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation Results

Dataset Perplexity
wikitext2 6.23
ptb 10.34
c4 9.15

Zero-shot accuracy (LM-Eval, %):

Task Accuracy
OpenBookQA 28.60
ARC-e 72.10
ARC-c 38.05
WinoGrande 66.61
HellaSwag 51.34
PIQA 75.52
Average 55.37

Rank Allocation

Per-layer ranks (click to expand)
Layer Rank
model.layers.0.mlp.gate_proj 1184
model.layers.0.mlp.up_proj 1176
model.layers.0.self_attn.k_proj 624
model.layers.0.self_attn.o_proj 1336
model.layers.0.self_attn.q_proj 632
model.layers.1.mlp.gate_proj 2072
model.layers.1.mlp.up_proj 1960
model.layers.1.self_attn.k_proj 632
model.layers.1.self_attn.o_proj 1232
model.layers.1.self_attn.q_proj 624
model.layers.10.mlp.down_proj 1800
model.layers.10.mlp.gate_proj 1784
model.layers.10.self_attn.k_proj 824
model.layers.10.self_attn.q_proj 800
model.layers.11.mlp.gate_proj 1816
model.layers.11.self_attn.k_proj 800
model.layers.11.self_attn.q_proj 752
model.layers.12.mlp.gate_proj 1720
model.layers.12.mlp.up_proj 2160
model.layers.12.self_attn.k_proj 768
model.layers.12.self_attn.q_proj 632
model.layers.13.mlp.gate_proj 1840
model.layers.13.self_attn.k_proj 776
model.layers.13.self_attn.o_proj 1568
model.layers.13.self_attn.q_proj 744
model.layers.14.mlp.gate_proj 2008
model.layers.14.self_attn.k_proj 824
model.layers.14.self_attn.q_proj 800
model.layers.15.mlp.gate_proj 2008
model.layers.15.self_attn.k_proj 952
model.layers.15.self_attn.q_proj 792
model.layers.16.self_attn.k_proj 640
model.layers.16.self_attn.q_proj 728
model.layers.17.mlp.gate_proj 2096
model.layers.17.self_attn.k_proj 696
model.layers.17.self_attn.q_proj 936
model.layers.18.mlp.gate_proj 2096
model.layers.18.self_attn.k_proj 616
model.layers.18.self_attn.q_proj 720
model.layers.19.mlp.gate_proj 1912
model.layers.19.self_attn.k_proj 696
model.layers.19.self_attn.q_proj 624
model.layers.2.mlp.gate_proj 1936
model.layers.2.self_attn.k_proj 664
model.layers.2.self_attn.q_proj 768
model.layers.20.self_attn.k_proj 768
model.layers.20.self_attn.o_proj 1360
model.layers.20.self_attn.q_proj 992
model.layers.20.self_attn.v_proj 1328
model.layers.21.mlp.gate_proj 2000
model.layers.21.mlp.up_proj 2008
model.layers.21.self_attn.k_proj 736
model.layers.21.self_attn.q_proj 712
model.layers.22.mlp.gate_proj 1912
model.layers.22.mlp.up_proj 1944
model.layers.22.self_attn.k_proj 632
model.layers.22.self_attn.o_proj 816
model.layers.22.self_attn.q_proj 664
model.layers.22.self_attn.v_proj 832
model.layers.23.mlp.gate_proj 2024
model.layers.23.mlp.up_proj 1984
model.layers.23.self_attn.k_proj 656
model.layers.23.self_attn.o_proj 1296
model.layers.23.self_attn.q_proj 616
model.layers.23.self_attn.v_proj 1160
model.layers.24.self_attn.k_proj 616
model.layers.24.self_attn.o_proj 1400
model.layers.24.self_attn.q_proj 704
model.layers.24.self_attn.v_proj 1168
model.layers.25.self_attn.k_proj 656
model.layers.25.self_attn.o_proj 848
model.layers.25.self_attn.q_proj 680
model.layers.25.self_attn.v_proj 1088
model.layers.26.self_attn.k_proj 888
model.layers.26.self_attn.o_proj 1104
model.layers.26.self_attn.q_proj 944
model.layers.26.self_attn.v_proj 1024
model.layers.27.self_attn.k_proj 656
model.layers.27.self_attn.o_proj 680
model.layers.27.self_attn.q_proj 616
model.layers.27.self_attn.v_proj 736
model.layers.28.self_attn.k_proj 616
model.layers.28.self_attn.o_proj 808
model.layers.28.self_attn.q_proj 696
model.layers.28.self_attn.v_proj 1128
model.layers.29.self_attn.k_proj 744
model.layers.29.self_attn.o_proj 1000
model.layers.29.self_attn.q_proj 616
model.layers.29.self_attn.v_proj 1296
model.layers.3.self_attn.k_proj 720
model.layers.3.self_attn.q_proj 784
model.layers.30.self_attn.k_proj 616
model.layers.30.self_attn.o_proj 776
model.layers.30.self_attn.q_proj 624
model.layers.30.self_attn.v_proj 992
model.layers.31.self_attn.k_proj 720
model.layers.31.self_attn.o_proj 728
model.layers.31.self_attn.q_proj 656
model.layers.31.self_attn.v_proj 800
model.layers.4.self_attn.k_proj 840
model.layers.4.self_attn.q_proj 784
model.layers.5.self_attn.k_proj 848
model.layers.5.self_attn.q_proj 880
model.layers.6.self_attn.k_proj 992
model.layers.6.self_attn.q_proj 848
model.layers.7.self_attn.k_proj 776
model.layers.7.self_attn.q_proj 696
model.layers.8.mlp.gate_proj 1896
model.layers.8.self_attn.k_proj 1176
model.layers.8.self_attn.q_proj 1024
model.layers.9.mlp.down_proj 2064
model.layers.9.mlp.gate_proj 1904
model.layers.9.self_attn.k_proj 912
model.layers.9.self_attn.q_proj 856

Hydra Configuration Summary

Config Field Value
Model jeffwan/llama-7b-hf
SVD Method kfac_svd
Search Method lems
Compression Target 0.8
Target Metric params
Calibration Dataset wikitext2
Calibration Samples 256
Sequence Length 2048
Seed 3
Downloads last month
22
Safetensors
Model size
5B params
Tensor type
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8

Finetuned
(5)
this model

Dataset used to train MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8

Collection including MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8

Evaluation results