Instructions to use MoritzMo123/kfac-svd_lems_llama-3-8b_0.9 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use MoritzMo123/kfac-svd_lems_llama-3-8b_0.9 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="MoritzMo123/kfac-svd_lems_llama-3-8b_0.9", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("MoritzMo123/kfac-svd_lems_llama-3-8b_0.9", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use MoritzMo123/kfac-svd_lems_llama-3-8b_0.9 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "MoritzMo123/kfac-svd_lems_llama-3-8b_0.9" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MoritzMo123/kfac-svd_lems_llama-3-8b_0.9", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/MoritzMo123/kfac-svd_lems_llama-3-8b_0.9
- SGLang
How to use MoritzMo123/kfac-svd_lems_llama-3-8b_0.9 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "MoritzMo123/kfac-svd_lems_llama-3-8b_0.9" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MoritzMo123/kfac-svd_lems_llama-3-8b_0.9", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "MoritzMo123/kfac-svd_lems_llama-3-8b_0.9" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MoritzMo123/kfac-svd_lems_llama-3-8b_0.9", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use MoritzMo123/kfac-svd_lems_llama-3-8b_0.9 with Docker Model Runner:
docker model run hf.co/MoritzMo123/kfac-svd_lems_llama-3-8b_0.9
unsloth/llama-3-8b — lems search — 0.9 target ratio
This model was compressed using kfac_svd with lems rank search starting from unsloth/llama-3-8b as base model. You may check out the paper, our publication, project page, and code repository for details on kfac-svd and our LEMS rank search.
Update (2026-10-23). The weights were recomputed with the fixed KFAC-SVD implementation. The first release of the code (v1, used for the published paper) whitened the output side of each weight with the Cholesky factor L of G = LLᵀ instead of Lᵀ. The correct version was described by the algorithm in the paper already. Those weights, which the paper's tables report, remain available as revision
v1-paper:AutoModelForCausalLM.from_pretrained("MoritzMo123/kfac-svd_lems_llama-3-8b_0.9", revision="v1-paper", trust_remote_code=True).
Compression Details
| Metric | Value |
|---|---|
| Base Model | unsloth/llama-3-8b |
| Method | kfac_svd |
| Search Method | lems |
| Target Ratio | 0.9 |
| Compression Metric | params |
| Recommended Dtype | float16 |
| Compressed Layers | 81 |
| Total Parameters | 7,332,196,433 |
Usage
The checkpoint records its recommended dtype in config.json; no explicit torch_dtype argument should be needed with this remote-code wrapper. For standard Transformers models, torch_dtype="auto" is the portable fallback.
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"MoritzMo123/kfac-svd_lems_llama-3-8b_0.9",
trust_remote_code=True,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("MoritzMo123/kfac-svd_lems_llama-3-8b_0.9")
inputs = tokenizer('Hello, ', return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Evaluation Results
| Dataset | Perplexity |
|---|---|
| wikitext2 | 6.51 |
| ptb | 10.94 |
| c4 | 10.90 |
Zero-shot accuracy (LM-Eval, %):
| Task | Accuracy |
|---|---|
| OpenBookQA | 33.60 |
| ARC-e | 80.35 |
| ARC-c | 49.32 |
| WinoGrande | 72.30 |
| HellaSwag | 57.42 |
| PIQA | 78.02 |
| Average | 61.83 |
Rank Allocation
Per-layer ranks (click to expand)
| Layer | Rank |
|---|---|
model.layers.0.self_attn.k_proj |
288 |
model.layers.0.self_attn.q_proj |
632 |
model.layers.1.self_attn.k_proj |
488 |
model.layers.1.self_attn.q_proj |
640 |
model.layers.10.mlp.down_proj |
2288 |
model.layers.10.mlp.gate_proj |
1768 |
model.layers.10.self_attn.k_proj |
752 |
model.layers.10.self_attn.q_proj |
712 |
model.layers.11.mlp.down_proj |
2168 |
model.layers.11.mlp.gate_proj |
1664 |
model.layers.11.self_attn.o_proj |
1488 |
model.layers.11.self_attn.q_proj |
880 |
model.layers.12.mlp.gate_proj |
1744 |
model.layers.12.self_attn.k_proj |
640 |
model.layers.12.self_attn.q_proj |
632 |
model.layers.13.mlp.gate_proj |
1912 |
model.layers.13.self_attn.k_proj |
624 |
model.layers.13.self_attn.q_proj |
808 |
model.layers.14.self_attn.q_proj |
784 |
model.layers.15.self_attn.k_proj |
736 |
model.layers.15.self_attn.q_proj |
728 |
model.layers.16.self_attn.k_proj |
728 |
model.layers.16.self_attn.q_proj |
632 |
model.layers.17.self_attn.k_proj |
720 |
model.layers.17.self_attn.q_proj |
656 |
model.layers.18.self_attn.k_proj |
576 |
model.layers.18.self_attn.q_proj |
720 |
model.layers.19.self_attn.k_proj |
656 |
model.layers.19.self_attn.o_proj |
1176 |
model.layers.19.self_attn.q_proj |
624 |
model.layers.2.self_attn.o_proj |
1216 |
model.layers.2.self_attn.q_proj |
800 |
model.layers.20.self_attn.k_proj |
504 |
model.layers.20.self_attn.o_proj |
960 |
model.layers.20.self_attn.q_proj |
624 |
model.layers.21.self_attn.k_proj |
568 |
model.layers.21.self_attn.o_proj |
944 |
model.layers.21.self_attn.q_proj |
648 |
model.layers.22.self_attn.k_proj |
592 |
model.layers.22.self_attn.o_proj |
1120 |
model.layers.22.self_attn.q_proj |
704 |
model.layers.23.self_attn.k_proj |
488 |
model.layers.23.self_attn.o_proj |
928 |
model.layers.23.self_attn.q_proj |
680 |
model.layers.24.self_attn.k_proj |
520 |
model.layers.24.self_attn.o_proj |
952 |
model.layers.24.self_attn.q_proj |
616 |
model.layers.25.self_attn.k_proj |
544 |
model.layers.25.self_attn.o_proj |
1024 |
model.layers.25.self_attn.q_proj |
624 |
model.layers.26.self_attn.k_proj |
424 |
model.layers.26.self_attn.o_proj |
952 |
model.layers.26.self_attn.q_proj |
616 |
model.layers.27.self_attn.k_proj |
416 |
model.layers.27.self_attn.o_proj |
912 |
model.layers.27.self_attn.q_proj |
664 |
model.layers.28.self_attn.k_proj |
744 |
model.layers.28.self_attn.o_proj |
1208 |
model.layers.28.self_attn.q_proj |
680 |
model.layers.29.self_attn.o_proj |
752 |
model.layers.29.self_attn.q_proj |
664 |
model.layers.3.self_attn.q_proj |
864 |
model.layers.30.self_attn.k_proj |
360 |
model.layers.30.self_attn.o_proj |
928 |
model.layers.30.self_attn.q_proj |
640 |
model.layers.31.self_attn.k_proj |
472 |
model.layers.31.self_attn.o_proj |
1264 |
model.layers.31.self_attn.q_proj |
648 |
model.layers.4.self_attn.o_proj |
1280 |
model.layers.4.self_attn.q_proj |
936 |
model.layers.5.self_attn.o_proj |
1200 |
model.layers.5.self_attn.q_proj |
784 |
model.layers.6.self_attn.q_proj |
712 |
model.layers.7.self_attn.k_proj |
768 |
model.layers.7.self_attn.q_proj |
688 |
model.layers.8.mlp.gate_proj |
2112 |
model.layers.8.self_attn.k_proj |
680 |
model.layers.8.self_attn.q_proj |
928 |
model.layers.9.mlp.gate_proj |
2128 |
model.layers.9.self_attn.o_proj |
1600 |
model.layers.9.self_attn.q_proj |
616 |
Hydra Configuration Summary
| Config Field | Value |
|---|---|
| Model | unsloth/llama-3-8b |
| SVD Method | kfac_svd |
| Search Method | lems |
| Compression Target | 0.9 |
| Target Metric | params |
| Calibration Dataset | wikitext2 |
| Calibration Samples | 256 |
| Sequence Length | 2048 |
| Seed | 42 |
- Downloads last month
- 217
Model tree for MoritzMo123/kfac-svd_lems_llama-3-8b_0.9
Base model
unsloth/llama-3-8bDataset used to train MoritzMo123/kfac-svd_lems_llama-3-8b_0.9
Collection including MoritzMo123/kfac-svd_lems_llama-3-8b_0.9
Paper for MoritzMo123/kfac-svd_lems_llama-3-8b_0.9
Evaluation results
- Perplexity on WikiText-2test set self-reported6.510
- Perplexity on PTBtest set self-reported10.940
- Perplexity on C4validation set self-reported10.900
- Accuracy on OpenBookQA (zero-shot)test set self-reported33.600
- Accuracy on ARC-e (zero-shot)test set self-reported80.350
- Accuracy on ARC-c (zero-shot)test set self-reported49.320
- Accuracy on WinoGrande (zero-shot)validation set self-reported72.300
- Accuracy on HellaSwag (zero-shot)validation set self-reported57.420