Receipt to JSON: on-device vision models

Small vision-language models that turn a photo of a shopping receipt into structured JSON (store, date, line items with quantity and price, tax, total). They are converted to LiteRT-LM .litertlm bundles that run fully offline on an Android phone's CPU, e.g. through the Flutter plugin flutter_edge_ai.

Recommended file: Qwen3.5-0.8B_receipts-QAT-v2_int8_1024.litertlm (1.32 GB).


Task	receipt photo → JSON (one image chunk per 1024 px of receipt height)
Base model	Qwen3.5-0.8B (vision + hybrid GatedDeltaNet / attention decoder)
Format	LiteRT-LM bundle, int8 weights, GatedDeltaNet decay/beta gates in float
Runtime	LiteRT-LM 0.18 (CPU; vision on GPU also works)
Languages seen in training	mostly German, English and Arabic receipts
Score	86.4 / 100 on 192 held-out receipts (project metric, see below)

Files

File	What it is	Status
`Qwen3.5-0.8B_receipts-QAT-v2_int8_1024.litertlm`	0.8B, trained on the relabelled dataset v2 (with `tax` field), quantization-aware, int8 + float gates	current, recommended
`Qwen3.5-0.8B_receipts-QAT_int8_1024.litertlm`	same recipe on the original labels (no `tax` field)	previous version
`Qwen3.5-0.8B_receipts-VL_mixed_1024.litertlm`	0.8B, no quantization-aware training, mixed int8/float	superseded
`Qwen3.5-0.8B_receipts_mixed.litertlm`	first 0.8B build	superseded
`Qwen3.5-2B_receipts-VL_mixed_1024.litertlm`	2B fine-tune, mixed int8/float (4.3 GB)	experimental, older labels
`OvisOCR2-receipts_mixed_1024.litertlm`	fine-tune of OvisOCR2 (a Qwen3.5-0.8B OCR model), see its `.md`	experimental, older labels
`model.safetensors`, `config.json`, tokenizer files	an early text-only Qwen3.5 checkpoint (`Qwen3_5ForCausalLM`), not the vision models above	kept for reference

The superseded files are kept so that existing app versions keep working.

Output format

One JSON object, keys in this order, every value a string:

{"store": "HELLWEG",
 "store_misc": "GmbH & Co. KG, Alt Biesdorf 40, 12683 Berlin-Biesdorf, Ust-ID: DE359540540, www.hellweg.de",
 "date_time": "08.06.2024 09:37:07",
 "articles": [
   {"name": "Classico Color 21", "misc": "15384, 40 08789 13170 6", "quantity": "1", "price": "22.99"},
   {"name": "Rasensand 20 kg", "misc": "840473, @ 8.99", "quantity": "2", "price": "17.98"}],
 "tax": "6.74",
 "total": "40.97"}

(Illustrative example, not a real model output.)

Conventions of the v2 model:

price is what the line costs in total (quantity × unit price), after a discount printed inside the line. It is not the unit price; the unit price goes into misc as @ x.xx when the quantity is not 1.
quantity is a count or a weight ("2", "0.938"), "1" if none is printed.
A discount or coupon printed as its own line is its own article with a negative price. Deposit, delivery, service and rounding lines are articles too, named as printed.
Payment lines, subtotals, tax summaries and totals are not articles; a wallet balance or voucher used for paying is not an article either.
tax is the total tax printed (several rates added up), "" if none is printed.
total is the final amount of the purchase incl. tax and fees, after discounts.
store is the brand; store_misc holds legal name, address, phone, tax / registration numbers, branch and website (no cashier, POS, order or payment details).
date_time is copied as printed and not normalised.

The older files (without -v2) use the same keys without tax, and their price mixes unit prices and line totals.

How to run

Input preparation (the model was trained exactly this way):

Resize the receipt photo to 1024 px width.
Cut it into 1024 px tall chunks with 200 px overlap.
Letterbox every chunk onto a white 1024 × 1024 square (top-left).
Send all chunks first and then the instruction text, in one user message, in a fresh conversation.

Instruction text (used in training; send it unchanged):

Look at the provided Image Chunks of the single Receipt and start extracting into the following JSON Format:
{
  "store":"",
  "store_misc":"",
  "date_time":"",
  "articles":[
    {
      "name":"",
      "misc":"",
      "quantity":"",
      "price":""
    }
  ],
  "total":""
}

Field rules:
- store: Only the name of the store
- store_misc: Additional store details like address
- date_time: Date and time from the receipt
- articles[].name: Only the product name
- articles[].misc: Article number or non-name details
- articles[].quantity: Quantity (use "1" if not shown)
- articles[].price: Decimal price number WITHOUT currency symbol
- total: Total sum of the invoice

It is very important to list every article.

Extract all information from this receipt image and return it as a JSON object.

Flutter (flutter_edge_ai):

await FlutterEdgeAi.installModel(modelType: ModelType.qwen35, fileType: ModelFileType.litertlm)
    .fromFile(path).install();
final model = await FlutterEdgeAi.getActiveModel(maxTokens: 8192, preferredBackend: PreferredBackend.cpu);

Python (litert_lm, desktop):

import litert_lm
engine = litert_lm.Engine("Qwen3.5-0.8B_receipts-QAT-v2_int8_1024.litertlm",
                          backend=litert_lm.Backend.CPU(), vision_backend=litert_lm.Backend.CPU(),
                          max_num_tokens=8192, max_num_images=8)
conv = engine.create_conversation(sampler_config=litert_lm.SamplerConfig(top_k=1, temperature=0.0),
                                  max_output_tokens=2048)
content = [{"type": "image", "path": p} for p in chunk_paths] + [{"type": "text", "text": PROMPT}]
reply = conv.send_message({"role": "user", "content": content})

Use greedy decoding (top_k 1). Parse the reply leniently: about 2 % of answers are not valid JSON (mostly repetition loops on very long receipts), and numbers can occasionally contain a stray comma.

Evaluation

Project scorer (0–100), averaging parse rate, store, date/time, total, article count and per-article name / quantity / price matches. Held-out set: 192 receipts (92 of the author's own German receipts, 100 from public receipt datasets), never used for training. All scores below are against the visually reviewed v2 reference labels.

Model	192 eval (PyTorch)	92 own receipts	`.litertlm` on 24 receipts (PC)
0.8B QAT v2	86.4	85.0	84.2
0.8B QAT (previous)	78.0	77.7	76.2
0.8B without QAT	77.3	–	–
2B	78.4	–	–

Field accuracy of the 0.8B QAT v2 model on the 192 receipts: total 96 %, store 92 %, date/time 86 %, article count 89 %, quantity 98 %, price 97 %, article name 80 %.

Speed on a phone

Samsung Galaxy S22 Ultra, 1.32 GB int8 bundle (measured with the previous 0.8B QAT bundle, which has the same architecture, size and recipe):

Backend	Load	First token, per image chunk	Decode	Typical receipt
CPU	23 s	38 s	9.5 tok/s	3.2 min
GPU vision + CPU text	25 s	32 s	12.1 tok/s	2.7 min
GPU	1.8 min	64 s	3.0 tok/s	7.7 min

About two thirds of the time goes into reading the image chunks, one third into writing the JSON.

Training

Full fine-tune of Qwen3.5-0.8B (vision tower and decoder; token embeddings frozen), 3 epochs, lr 1e-5, effective batch 16, bf16 autocast with fp32 master weights, on one RTX PRO 6000.
Trained in the on-device input format: square 1024 px chunks and plain 1-D positions for all tokens (what LiteRT-LM feeds), instead of Qwen's 2-D image positions.
Quantization-aware training: every linear layer of the decoder computes with fake int8 (weights per output channel, activations per token), matching the dynamic int8 kernels on the phone. The GatedDeltaNet decay/beta gates (in_proj_a, in_proj_b) stay float, in training and in the bundle: int8 there made long article lists go missing (24-receipt test: 62 → 77).
Augmentation: the author's own receipts are repeated 3× per epoch, and 2/3 of those draws become a random phone-capture variant (slight rotation and perspective, lighting and shadows, blur, noise, JPEG). Text stays readable, so the labels stay valid.
Data (dataset v2): about 3,800 training receipts: own German receipt photos plus public receipt datasets (Egyptian / Middle-Eastern receipts and scanned Malaysian receipts). Labels were re-derived with a 27B vision model and then checked against the images by reviewers, including every eval receipt and about 750 suspicious training receipts; 62 receipts with unsure labels were left out. The dataset is not published (it contains private receipts).

Limitations

Long receipts (40+ lines) take minutes on a phone CPU and occasionally end in a repetition loop.
Faded thermal prints, folds and strong perspective cause digit errors; the model does not check that the article prices add up. A sum check in the app is recommended.
Trained mostly on German retail and Egyptian / Middle-Eastern receipts; other countries, handwritten bills and invoices are out of scope.
date_time is not normalised; parse it leniently.