Cukinator/Mistral-7B-4bit-Ignacio-Final

Mistral 7B v0.1 — 4-bit NF4

A BitsAndBytes 4-bit NF4 version of Mistral-7B-v0.1 for text generation experiments. The source notebook loads the original base model, quantizes it, and saves it. It does not perform supervised fine-tuning, LoRA training or teacher distillation.

The underlying Mistral architecture has 7,241,732,096 parameters (7.24B) and appears as 7B in Hugging Face listings. Four-bit refers to the quantization of the linear weights, not the parameter count, compute precision, or precision of every tensor.

Model and quantization

Property	Value
Task	Text Generation / text completion
Base model	mistralai/Mistral-7B-v0.1
Architecture	MistralForCausalLM; 32 layers; hidden size 4,096
Release files	model.safetensors, model config and tokenizer
Quantization library / format	BitsAndBytes / 4-bit NF4
Double quantization	Enabled
Compute dtype / quantized storage	FP16 / uint8
Configured context / sliding window	32,768 positions / 4,096
Source hardware	Two Tesla T4 GPUs; layers distributed with device_map="balanced"

This is the base completion model, not Mistral Instruct. The uploaded tokenizer has no chat template; use plain text prompts.

The published config includes the BitsAndBytes quantization configuration. Quantization metadata, embeddings and other tensors add overhead beyond the packed linear weights.

Recorded evaluation

The source notebook reports:

Measurement	Saved result	Scope
WikiText-2 perplexity	4.7945213	Test split; custom sliding-window implementation
ARC-Challenge accuracy (acc)	49.0%	100 examples, zero-shot
ARC-Challenge length-normalized accuracy (acc_norm)	46.0%	Same 100-example evaluation
Generation throughput	About 10.9 tokens/s	Four prompts, three timed runs each, up to 128 new tokens
Allocated GPU memory	About 4.15 GB total	Sum of torch.cuda.memory_allocated across both GPUs

The perplexity implementation uses 2,048-token windows and a stride of 512, masks overlapping targets, then exponentiates the unweighted mean of window losses. This is its recorded estimator, not a token-weighted reference evaluation.

The memory reading is allocated memory at measurement time, not peak process memory or a guaranteed minimum GPU requirement. Throughput depends on hardware, placement, prompt length and library versions.

The notebook's FP16 reference numbers are constants rather than a matched baseline run. Its comparisons do not establish that quantization improves perplexity or preserves full benchmark accuracy. The ARC result covers only 100 questions. Evaluation used the in-memory quantized base model; no Hub revision was pinned or reloaded to bind those results to this uploaded file. The notebook also uses a local output name without the final "-Final" suffix.

Quick start

Use a CUDA-enabled PyTorch installation and a compatible BitsAndBytes build:

pip install -U torch "transformers>=4.57.1,<5" accelerate bitsandbytes sentencepiece

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Cukinator/Mistral-7B-4bit-Ignacio-Final"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype=torch.float16,
    device_map="auto",
)
model.eval()

inputs = tokenizer(
    "Explain the theory of relativity in simple terms:",
    return_tensors="pt",
).to(model.get_input_embeddings().weight.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(
    output[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
))

The saved config supplies the quantization settings; the example loads the existing quantized checkpoint. It does not apply a new quantizer to the uploaded weights. The source was run on CUDA/T4 hardware; compatibility with other backends was not verified in this review.

Intended use and limitations

Use this release to explore quantized text completion and inference memory tradeoffs. A base language model can continue a prompt without following instructions reliably. It can generate incorrect information or reproduce biases from the upstream model. The saved benchmarks are limited experiments and do not certify deployment quality.

Sources and license

• Quantization and benchmark notebook, pinned to source commit 380b833d10eaaf39bfdeb6b09dfb1bf6dec2f269.
• Published model config.
• Upstream Mistral model card.
• Project repository.

Model license: Apache 2.0, inherited from the upstream model. Reviewed on October 9, 2026 against the source notebook and uploaded metadata; no new inference or benchmark run is claimed.