sahilchachra/LensVLM-9B-MXFP8

LensVLM-9B — MLX MXFP8

MLX MXFP8 quantization of apple/LensVLM-9B, Apple's document-QA vision-language model built on Qwen3.5-9B. LensVLM is a qwen3_5 VLM with a hybrid linear-attention (GatedDeltaNet) + full-attention text backbone (32 layers, full attention every 4th layer) and a SigLIP-style vision tower, trained to scan compressed page thumbnails and selectively expand only the relevant pages via tool calls — an efficient approach to long, image-heavy document understanding. Runs on Apple Silicon via mlx-vlm. Stays image-text-to-text — the vision tower is kept in bf16; only the text backbone is quantized.

Precision	MXFP8 (E4M3 + E8M0 shared scale, group size 32)
Bits per weight	8.626 bpw
On-disk size	9.5 GB (2 shards)
Quantized	text backbone (hybrid linear/full attention, incl. lm_head)
Kept in bf16	vision tower (SigLIP-style, deepstack_visual_indexes=[])

Quantizations

Variant	Bits	Size
LensVLM-9B-MXFP4	4.821 bpw	5.3 GB	smaller / for 8 GB+ Macs
LensVLM-9B-MXFP8	8.626 bpw	9.5 GB	← this repo

Verification

Both builds were tested end-to-end on Apple Silicon (native mlx-vlm and LM Studio 0.4.20 / mlx-llm runtime), with deterministic greedy decoding, inspecting raw token IDs (not just detokenized text) to rule out plausible-looking-but-garbage output:

Text (native mlx-vlm) — coherent, on-architecture reasoning (LensVLM answers as a document-scanning agent, referencing "pages" / "tools" even with no document attached — this is expected trained behavior, not a quantization artifact):

• "What is the capital of France?" → coherent reasoning about locating a relevant page/section.
• "What is 25 + 17?" → correct step-by-step arithmetic reasoning.

Text (LM Studio API, /v1/chat/completions) — same document-scanning-agent behavior, followed through to a correct final answer:

• "What is the capital of France?" (500 max_tokens) → the model simulates scanning page thumbnails, "reads" pages 2, 5, 6, and correctly concludes "Paris".
• "What is 25 + 17?" → "42".

Vision (bf16 vision tower, through the quantized text backbone) — tested both via mlx-vlm and via LM Studio's base64 image API:

Image	Question	Answer
solid red	main color?	"red"	✅
solid blue	main color?	"blue"	✅
green circle	shape and color?	"green circle"	✅

Color and shape are read correctly on both MXFP4 and MXFP8 — the vision path is live end-to-end, including through the LM Studio OpenAI-compatible API.

Usage (mlx-vlm)

pip install -U mlx-vlm   # needs the qwen3_5 architecture (>= 0.6.12)

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("sahilchachra/LensVLM-9B-MXFP8")
config = model.config

prompt = apply_chat_template(processor, config, "What is the capital of France?")
print(generate(model, processor, prompt, max_tokens=300, verbose=True))

For image input, pass image=<path> to generate(). This is a reasoning/agentic model: it emits a <think>-like reasoning channel that references document pages and tools before its final answer, so give it enough max_tokens (300+) or the final answer can be cut off mid-reasoning.

Run in LM Studio

Loads and runs in LM Studio (tested on 0.4.20+, mlx-llm runtime): the qwen3_5 architecture is recognized, the model indexes cleanly (format: mlx metadata present in the safetensors shards), and the default ChatML-style template runs as-is. Both text and image inputs work correctly through the /v1/chat/completions API (verified above) — reasoning content streams into reasoning_content, final answer into content. Needs a 16 GB+ Mac to load comfortably under LM Studio's default guardrails.

Notes & limitations

• Document-scanning-agent behavior by design. LensVLM was trained to operate over compressed multi-page documents with tool-calling ("expand this page"). Without an actual document/tool context, it will still narrate a page-scanning reasoning trace before answering from its own knowledge — this is expected base-model behavior carried through quantization faithfully, not a bug.
• tie_word_embeddings=false; the lm_head is quantized along with the rest of the text backbone.
• Inherits all capabilities and limitations of the base model, and its license (Apple Machine Learning Research license — see LICENSE, NOTICE, ACKNOWLEDGEMENTS, all reproduced in this repo).
• Quantized by @sahilchachra with MLX.