Falcon-H1-0.5B-Instruct — LiteRT-LM

tiiuae/Falcon-H1-0.5B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15. To our knowledge this is the first Falcon-H1 in LiteRT form, and the first fully-hybrid (parallel attention + Mamba2 SSM in every layer) architecture served by the released runtime — including on the GPU.

Falcon-H1 is TII's hybrid design: every one of the 36 layers runs a grouped-query attention branch and a Mamba2 selective-scan branch in parallel on the same input and sums them. Each layer therefore carries both a KV cache and constant-size conv + SSM recurrent state.

File	Recipe	Size
Falcon-H1-0.5B-Instruct_int8.litertlm	int8 dynamic on linears + embedding (convs and the scan stay float); fp32 activations declared for GPU	650 MB

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

Correctness

• Logits parity vs PyTorch: the float export matches the HF model teacher-forced across 8 decode positions — max|logit diff| 7.6e-05, correlation 1.000000, top-1 and top-5 identical at every position.
• 8-question sanity gate: int8 = float = GPU — all three answer 6/8 with near-verbatim identical text; the two misses ("merci" and 8×7) are the 0.5B model's own level (the float graph misses them the same way), not conversion or quantization damage.
• Prompt-length robustness: hermetic prefill-chunk sweep (fresh engine per length, 12–60 tokens) — 41/41 clean.
• Devices: iPhone 17 Pro (Metal) runs the composite quality probe at 7/8; Pixel 8a (OpenCL) delegates every subgraph fully (e.g. 5366/5366, zero rejections) with correct output.

Usage

litert-lm run ./Falcon-H1-0.5B-Instruct_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Falcon-H1-0.5B-Instruct_int8.litertlm --backend gpu --cache no --prompt "..."

Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks. The bundle carries the tokenizer and the stock ChatML-style Falcon-H1 chat template.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	2650 tok/s	127.5 tok/s	0.10 s
CPU	473 tok/s	59.0 tok/s	0.56 s

On device (cold start, single runs, 145-token composite prompt, quality harness):

Device	Backend	Prefill	Decode	TTFT	Peak memory
iPhone 17 Pro	GPU (Metal)	365.9 tok/s	52.7 tok/s	0.46 s	2.50 GB
iPhone 17 Pro	CPU	332.3 tok/s	36.2 tok/s	0.48 s	0.66 GB

Pixel 8a (Tensor G3, litert_lm_main built from the v0.16.0 tag, OpenCL, default 19-token prompt): TTFT 0.93 s, prefill 23.6 tok/s, decode 11.8 tok/s, full delegation. The CPU on the same Pixel 8a (litert_lm_advanced_main from litert-lm 0.16.0, 205-token prompt, two cold runs, 2026-09-05) decodes 22.5–25.9 tok/s (prefill 89.6–98.7 tok/s, TTFT 2.19–2.41 s, engine init 4.5 s, peak 1374 MB), about twice the GPU figure. The two runs differ in binary and prompt length, so that ordering is indicated, not measured under one protocol; the manifest recommends cpu on 8 GB-class Android until a same-binary GPU run settles it.

Honest notes:

• Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal), Pixel 8a (Arm Mali, OpenCL) and Qualcomm Adreno — the Galaxy S26 section below is that measurement.
• On low-end Android the GPU buys prefill and time-to-first-token, not decode (decode is memory-bandwidth-bound there; the CPU path reads int8 weights while the fp32-activation GPU path reads expanded ones). Pick the backend for your workload: long prompts favour the GPU, long answers favour the CPU. On Apple hardware the GPU wins across the board.
• GPU runs with fp32 activations (declared in the bundle) — that is the GPU memory multiple above (2.50 GB vs 0.66 GB on iPhone).
• At 0.5B the model itself is weak at arithmetic and non-English trivia; int8 adds borderline greedy flips on exactly those items (e.g. 8×7 differs between backends). Everything stays coherent — there is no degeneracy.

Galaxy S26 — GPU and CPU

The published bundle runs on the Android GPU backend: LiteRT takes the whole graph and the model generates.

file	GPU backend	delegation	peak
Falcon-H1-0.5B-Instruct_int8.litertlm	runs	59096 / 59096 ops across 12 subgraphs on LiteRT GPU	2613 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

Speed on the same handset with one binary and one 205-token prompt file; only --backend changes (litert_lm_advanced_main from litert-lm 0.16.0, --benchmark, --sampler_backend=cpu). The GPU row is the gate run of 2026-08-24 (one run); the CPU rows were taken 2026-09-05, two cold runs (compile and weight caches deleted between runs) with the phone cooled below 42 °C before each run.

Backend	Prefill	Decode	TTFT	Engine init	Peak (VmHWM)
GPU (LiteRT GPU, full delegation)	386.7 tok/s	28.8 tok/s	0.58 s	11.3 s	2613 MB
CPU (XNNPACK)	217–317 tok/s	32.1–40.5 tok/s	0.69–1.01 s	1.8 s	1408 MB

The CPU decodes 1.3× faster, at half the peak memory and without the 11 s engine init; the GPU wins prefill. This repo's litertlm_manifest.json recommends cpu for flagship-class Android on that pair. Pick gpu for long prompts with short answers.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm falcon_h1_work/):

• Composite hybrid cache layer: every layer holds KV + conv + recurrent state at ONE layer index — a cache layer class that is full-attention and Mamba2 at the same time (the runtime binds states by tensor name, so co-residency is just packaging).
• Folded selective scan: the Mamba2 scan is re-expressed as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math) — this is what makes the graph fully delegable on GPU.
• Falcon-specific wiring: the µP multiplier vector (mup_vector, a non-persistent model-level buffer) and ssm_in_multiplier are preserved in the traced scan; the exporter's timestamp-index kwargs are re-injected at the attention layer (FalconH1's layer loop drops kwargs).
• Prefill-pad guard: the runtime runs partially-filled prefill chunks; pad positions are made exact identity steps for the SSM and the stored conv window is gathered at the last valid column.
• Quantization: post-hoc dynamic int8 over linears + embedding only; convs and the scan stay float.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
Falcon-H1-0.5B-Instruct_int8.litertlm	53.9 (53.4–54.7)	7.0 (7.0–7.2)	4.9 s	1.3 GB

License and changes

Distributed under the Falcon LLM License (inherited from the base model — see the license link). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified. This repository is a community conversion and is not affiliated with TII.