Instructions to use MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8
- SGLang
How to use MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8 with Docker Model Runner:
docker model run hf.co/MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8
jeffwan/llama-7b-hf — lems search — 0.8 target ratio
This model was compressed using kfac_svd with lems rank search starting from jeffwan/llama-7b-hf as base model. You may check out our publication and project page for details on kfac-svd and our LEMS rank search.
Note (2026-10-23). These weights were computed with the fixed KFAC-SVD implementation. The first release of the code (v1, used for the published paper) whitened the output side of each weight with the Cholesky factor L of G = LLᵀ instead of Lᵀ. The correct version was described by the algorithm in the paper already.
Compression Details
| Metric | Value |
|---|---|
| Base Model | jeffwan/llama-7b-hf |
| Method | kfac_svd |
| Search Method | lems |
| Target Ratio | 0.8 |
| Compression Metric | params |
| Recommended Dtype | float16 |
| Compressed Layers | 114 |
| Total Parameters | 5,443,037,298 |
Usage
The checkpoint records its recommended dtype in config.json; no explicit torch_dtype argument should be needed with this remote-code wrapper. For standard Transformers models, torch_dtype="auto" is the portable fallback.
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8",
trust_remote_code=True,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8")
inputs = tokenizer('Hello, ', return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Evaluation Results
| Dataset | Perplexity |
|---|---|
| wikitext2 | 6.23 |
| ptb | 10.34 |
| c4 | 9.15 |
Zero-shot accuracy (LM-Eval, %):
| Task | Accuracy |
|---|---|
| OpenBookQA | 28.60 |
| ARC-e | 72.10 |
| ARC-c | 38.05 |
| WinoGrande | 66.61 |
| HellaSwag | 51.34 |
| PIQA | 75.52 |
| Average | 55.37 |
Rank Allocation
Per-layer ranks (click to expand)
| Layer | Rank |
|---|---|
model.layers.0.mlp.gate_proj |
1184 |
model.layers.0.mlp.up_proj |
1176 |
model.layers.0.self_attn.k_proj |
624 |
model.layers.0.self_attn.o_proj |
1336 |
model.layers.0.self_attn.q_proj |
632 |
model.layers.1.mlp.gate_proj |
2072 |
model.layers.1.mlp.up_proj |
1960 |
model.layers.1.self_attn.k_proj |
632 |
model.layers.1.self_attn.o_proj |
1232 |
model.layers.1.self_attn.q_proj |
624 |
model.layers.10.mlp.down_proj |
1800 |
model.layers.10.mlp.gate_proj |
1784 |
model.layers.10.self_attn.k_proj |
824 |
model.layers.10.self_attn.q_proj |
800 |
model.layers.11.mlp.gate_proj |
1816 |
model.layers.11.self_attn.k_proj |
800 |
model.layers.11.self_attn.q_proj |
752 |
model.layers.12.mlp.gate_proj |
1720 |
model.layers.12.mlp.up_proj |
2160 |
model.layers.12.self_attn.k_proj |
768 |
model.layers.12.self_attn.q_proj |
632 |
model.layers.13.mlp.gate_proj |
1840 |
model.layers.13.self_attn.k_proj |
776 |
model.layers.13.self_attn.o_proj |
1568 |
model.layers.13.self_attn.q_proj |
744 |
model.layers.14.mlp.gate_proj |
2008 |
model.layers.14.self_attn.k_proj |
824 |
model.layers.14.self_attn.q_proj |
800 |
model.layers.15.mlp.gate_proj |
2008 |
model.layers.15.self_attn.k_proj |
952 |
model.layers.15.self_attn.q_proj |
792 |
model.layers.16.self_attn.k_proj |
640 |
model.layers.16.self_attn.q_proj |
728 |
model.layers.17.mlp.gate_proj |
2096 |
model.layers.17.self_attn.k_proj |
696 |
model.layers.17.self_attn.q_proj |
936 |
model.layers.18.mlp.gate_proj |
2096 |
model.layers.18.self_attn.k_proj |
616 |
model.layers.18.self_attn.q_proj |
720 |
model.layers.19.mlp.gate_proj |
1912 |
model.layers.19.self_attn.k_proj |
696 |
model.layers.19.self_attn.q_proj |
624 |
model.layers.2.mlp.gate_proj |
1936 |
model.layers.2.self_attn.k_proj |
664 |
model.layers.2.self_attn.q_proj |
768 |
model.layers.20.self_attn.k_proj |
768 |
model.layers.20.self_attn.o_proj |
1360 |
model.layers.20.self_attn.q_proj |
992 |
model.layers.20.self_attn.v_proj |
1328 |
model.layers.21.mlp.gate_proj |
2000 |
model.layers.21.mlp.up_proj |
2008 |
model.layers.21.self_attn.k_proj |
736 |
model.layers.21.self_attn.q_proj |
712 |
model.layers.22.mlp.gate_proj |
1912 |
model.layers.22.mlp.up_proj |
1944 |
model.layers.22.self_attn.k_proj |
632 |
model.layers.22.self_attn.o_proj |
816 |
model.layers.22.self_attn.q_proj |
664 |
model.layers.22.self_attn.v_proj |
832 |
model.layers.23.mlp.gate_proj |
2024 |
model.layers.23.mlp.up_proj |
1984 |
model.layers.23.self_attn.k_proj |
656 |
model.layers.23.self_attn.o_proj |
1296 |
model.layers.23.self_attn.q_proj |
616 |
model.layers.23.self_attn.v_proj |
1160 |
model.layers.24.self_attn.k_proj |
616 |
model.layers.24.self_attn.o_proj |
1400 |
model.layers.24.self_attn.q_proj |
704 |
model.layers.24.self_attn.v_proj |
1168 |
model.layers.25.self_attn.k_proj |
656 |
model.layers.25.self_attn.o_proj |
848 |
model.layers.25.self_attn.q_proj |
680 |
model.layers.25.self_attn.v_proj |
1088 |
model.layers.26.self_attn.k_proj |
888 |
model.layers.26.self_attn.o_proj |
1104 |
model.layers.26.self_attn.q_proj |
944 |
model.layers.26.self_attn.v_proj |
1024 |
model.layers.27.self_attn.k_proj |
656 |
model.layers.27.self_attn.o_proj |
680 |
model.layers.27.self_attn.q_proj |
616 |
model.layers.27.self_attn.v_proj |
736 |
model.layers.28.self_attn.k_proj |
616 |
model.layers.28.self_attn.o_proj |
808 |
model.layers.28.self_attn.q_proj |
696 |
model.layers.28.self_attn.v_proj |
1128 |
model.layers.29.self_attn.k_proj |
744 |
model.layers.29.self_attn.o_proj |
1000 |
model.layers.29.self_attn.q_proj |
616 |
model.layers.29.self_attn.v_proj |
1296 |
model.layers.3.self_attn.k_proj |
720 |
model.layers.3.self_attn.q_proj |
784 |
model.layers.30.self_attn.k_proj |
616 |
model.layers.30.self_attn.o_proj |
776 |
model.layers.30.self_attn.q_proj |
624 |
model.layers.30.self_attn.v_proj |
992 |
model.layers.31.self_attn.k_proj |
720 |
model.layers.31.self_attn.o_proj |
728 |
model.layers.31.self_attn.q_proj |
656 |
model.layers.31.self_attn.v_proj |
800 |
model.layers.4.self_attn.k_proj |
840 |
model.layers.4.self_attn.q_proj |
784 |
model.layers.5.self_attn.k_proj |
848 |
model.layers.5.self_attn.q_proj |
880 |
model.layers.6.self_attn.k_proj |
992 |
model.layers.6.self_attn.q_proj |
848 |
model.layers.7.self_attn.k_proj |
776 |
model.layers.7.self_attn.q_proj |
696 |
model.layers.8.mlp.gate_proj |
1896 |
model.layers.8.self_attn.k_proj |
1176 |
model.layers.8.self_attn.q_proj |
1024 |
model.layers.9.mlp.down_proj |
2064 |
model.layers.9.mlp.gate_proj |
1904 |
model.layers.9.self_attn.k_proj |
912 |
model.layers.9.self_attn.q_proj |
856 |
Hydra Configuration Summary
| Config Field | Value |
|---|---|
| Model | jeffwan/llama-7b-hf |
| SVD Method | kfac_svd |
| Search Method | lems |
| Compression Target | 0.8 |
| Target Metric | params |
| Calibration Dataset | wikitext2 |
| Calibration Samples | 256 |
| Sequence Length | 2048 |
| Seed | 3 |
- Downloads last month
- 22
Model tree for MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8
Base model
jeffwan/llama-7b-hfDataset used to train MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8
Collection including MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8
Evaluation results
- Perplexity on WikiText-2test set self-reported6.230
- Perplexity on PTBtest set self-reported10.340
- Perplexity on C4validation set self-reported9.150
- Accuracy on OpenBookQA (zero-shot)test set self-reported28.600
- Accuracy on ARC-e (zero-shot)test set self-reported72.100
- Accuracy on ARC-c (zero-shot)test set self-reported38.050
- Accuracy on WinoGrande (zero-shot)validation set self-reported66.610
- Accuracy on HellaSwag (zero-shot)validation set self-reported51.340