Usage agreement: free, non-commercial use only

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is shared freely for the sake of Allah. By requesting access you agree: (1) you may use, fine-tune and redistribute it and its outputs ONLY in applications that are FREE to end users; (2) you may NOT sell it, place it behind a paid subscription or paywall, monetize it with ads, or earn any revenue from an app or service that uses this model or its outputs; (3) these terms pass on to anyone you share it with. وَما أَسأَلُكُم عَلَيهِ مِن أَجرٍ إِن أَجرِيَ إِلّا عَلىٰ رَبِّ العالَمِين

zipformer_p-arabic-v2

A streaming phoneme recognizer for Arabic. It maps audio to phoneme units (consonant plus haraka) representing what was actually pronounced, including dialect and tajweed.

Scope. This is a general Arabic model. It covers everyday speech (MSA, broadcast, and Egyptian, Gulf, Levantine and Maghrebi dialect) as well as Qur'anic recitation with tajweed. Qur'an-only models typically degrade sharply on ordinary Arabic; this model is trained for both domains in one.

Comparison with the models it replaces

	this model	`zipformer_p-quran` (Qur'an specialist)	`zipformer_p-arabic` (v1, broad)
Qur'an, real phone audio (held-out PER)	8.14%	13.39%	34.72%
Qur'an, clean recitation (PER)	3.31%	5.82%	not applicable
General Arabic (khutbah, CER)	4.7%	22.4%	6.9%
General Arabic (khutbah, WER)	9.4%	36.9%	11.4%

The Qur'an specialist is accurate on clean recitation but loses ground on real phone audio, and its tajweed markers transfer poorly to everyday speech, which is what the general Arabic rows show. Training both domains jointly retained Qur'an accuracy and improved general Arabic.

Specification


Architecture	Zipformer2 (icefall/k2), causal streaming, CTC
Parameters	64.7M
Vocabulary	250 phoneme units plus blank
Features	80-bin kaldi fbank (povey), 16 kHz
Training	from scratch, 10 epochs, 2,039 h unique audio, last 3 epochs averaged
Streaming	`decode_chunk_len` 48, chunk 24, left context 256
Formats	PyTorch, ONNX fp32, ONNX int8, CoreML fp16 (ANE, split multifunction)

Benchmarks

Official Qur'an-Lab benchmark (600 clips)

Scored with the official score.py normalisation. Phonemes are converted to text by nearest-ayah retrieval, the same method used for the published zipformer_p-quran entry.

Metric	this model	`zipformer_p-quran`
Overall WER	5.64	5.83
Phone WER (`tlog_holdout`)	7.56	7.76
EveryAyah WER	7.72	7.64
QUL WER	2.19	2.66
WER (alef-insensitive)	4.40	4.59

Phoneme error rate

Test set	this model	`zipformer_p-quran`
Held-out real-phone Qur'an (793 clips, deterministic gold)	8.14%	13.39%
Clean Qur'an recitation (400 clips)	3.31%	5.82%
Benchmark clips with gold phonemes (492)	8.26%	11.54%

PER is the appropriate metric for this model, and the two tables measure different things. Nearest-ayah retrieval constrains output to one of about 9,112 ayat, so it corrects most acoustic errors before WER is computed: retrieval selects the correct ayah 99.8% of the time for this model against 99.6% for the specialist. With that little headroom, leaderboard WER compresses a 28% relative PER improvement into a 3% WER improvement. Use PER to compare acoustic quality and WER to compare end-to-end retrieval systems.

General Arabic (non-Qur'an)

Reverberant YouTube khutbah (MSA sermon), phonemes converted to text by an LLM, scored against ElevenLabs Scribe:

	WER	CER
this model	9.4%	4.7%
`zipformer_p-arabic` (v1)	11.4%	6.9%
`zipformer_p-quran`	36.9%	22.4%

Noise robustness

v1 was trained on clean, un-augmented audio. v2 applies three augmentations on GPU every batch, ramped in over the first 6,000 steps:

Augmentation	Setting	Purpose
MUSAN additive noise	1,983 clips (noise, music, babble), p=0.5, SNR 10 to 20 dB	rooms, traffic, crowds, PA systems, phone audio
Speed perturbation	0.9x, 1.0x, 1.1x	fast and slow delivery
SpecAugment	2 frequency masks (max 27 bins), 2 time masks (max 15% total)	reduces over-fitting to specific frames and bands

The effect is largest where conditions are worst. On 793 held-out real phone recordings v2 scores 8.14% PER against 13.39% for the Qur'an specialist, a 39% relative reduction. On clean studio recitation the margin narrows to 3.31% against 5.82%.

MUSAN is narrowband: median 99% energy bandwidth is about 3.2 kHz, with under 0.6% of energy above 6 kHz. The model is therefore well hardened against low and mid frequency interference, which covers most real-world noise, but has seen comparatively little high frequency hiss.

Training data

2,039 h of unique audio.

Broad Arabic, 1,527 h (MSA, broadcast, and dialect coverage)

Source	Hours	Share	Contribution
MASC	859.8 h	56.3%	dialectal YouTube speech (Egyptian, Gulf, Levantine)
SADA	254.7 h	16.7%	Saudi broadcast and drama, dialectal
MGB2	225.7 h	14.8%	Al Jazeera broadcast MSA
`mostafa_mahmoud`	67.2 h	4.4%	Egyptian colloquial broadcast speech
CommonVoice	64.8 h	4.2%	read MSA
ArVoice	49.1 h	3.2%	read and expressive
FLEURS	5.8 h	0.4%	read MSA

Qur'an, 512 h

Source	Hours	Share	Contribution
EveryAyah	392.0 h	76.6%	multi-reciter studio recitation
`tlog` (tier S)	85.2 h	16.6%	real-phone recitation: phones, rooms, background noise
muaalem	27.4 h	5.4%	tajweed-annotated recitation
`tlog` (tiers A and B)	6.8 h	1.3%	lower-confidence labels, excluded from this run

EveryAyah and muaalem are repeated 4x per epoch. The effective training mix is 3,290 h per epoch:

Component	Effective hours	Share of epoch
Clean studio Qur'an (EveryAyah)	1,568 h	47.7%
Broad Arabic	1,527 h	46.4%
Tajweed-annotated Qur'an (muaalem)	110 h	3.3%
Real-phone Qur'an (`tlog` tier S)	85 h	2.6%

Real-phone Qur'an is a small fraction of the mix, so the noise robustness reported above comes predominantly from the augmentation and from the acoustic variety of the broad Arabic corpora rather than from volume of in-domain noisy recitation.

For comparison, zipformer_p-quran was trained almost entirely on Qur'an, dominated by clean studio recitation. Applications tuned against that model may need their thresholds recalibrated for v2, since clean studio audio accounts for a smaller share of this model's training distribution.

Labels

The audio above comes from public and third-party corpora. The phoneme labelling is ours. No public Arabic phoneme corpus exists at this scale, so every phoneme label used in training was produced for this project and released as two datasets:

quran-phonemes, deterministic. Generated rule-based with quran-transcript (the generator behind the muaalem gold), so tajweed is exact rather than inferred: madd lengths, ghunna, idgham and qalqala follow Hafs rules.
arabic-broad-phonemes, LLM-phonemized and source-aware. Dialectal sources are prompted for dialectal vowels and MSA sources for MSA, preserving dialect rather than normalising it. Being LLM-generated it carries some noise, mainly vowel-length artifacts.
Quality tiering. The tlog real-phone clips are tiered S, A and B by teacher-label confidence and weighted accordingly, so noisier labels count for less.

The tlog_holdout evaluation set is excluded from training, so the real-phone figures above are measured on unseen clips of the same domain.

The 41-symbol Qur'an alphabet is a superset of the 37-symbol broad alphabet, so both are covered by one 250-unit tokenizer.

Files

File	Size	Contents
`zipformer_p_arabic_v2.pt`	259 MB	PyTorch weights (64.7M params) plus `blank_id`
`zipformer_p_arabic_v2.onnx`	263 MB	cache-aware streaming CTC, sherpa-onnx compatible
`zipformer_p_arabic_v2.int8.onnx`	73 MB	same graph, dynamic INT8, for mobile and edge
`zipformer_p_arabic_v2.float8.mlpackage`	63 MB	CoreML, 8-bit weights + fp16 compute, Apple Neural Engine, iOS 18+ (multifunction: `front` and `back`). Recommended.
`zipformer_p_arabic_v2.float16.mlpackage`	124 MB	CoreML fp16, same graph and interface as float8
`packing_front.json`, `packing_back.json`	2 KB	state blob layout for the CoreML packages (see the CoreML section)
`tokens.txt`	3 KB	symbol table for decoding
`phoneme_units.json`	3 KB	training-time tokenizer (greedy longest match)
`decode_with_confidence.py`		per-phoneme confidence, alternatives and timings
`LICENSE`		free / non-commercial terms

Symbol table

Decode with tokens.txt. It is authoritative and is the format sherpa-onnx expects.

phoneme_units.json is a training-time artifact whose indices do not describe the model's output layer. It numbers <blank> as 0 and the phonemes 1 to 250, whereas the model emits phonemes at 0 to 249 with blank at 250. A symbol table built from the JSON is therefore shifted by one symbol. The model loads, decodes and emits well-formed Arabic in that state, so the failure is silent and appears only as degraded accuracy. Thanks to the QuranTracker team for identifying and reporting this.

Blank at 250 is intentional and matches zipformer_p-quran. It will not change without a corresponding tokens.txt published alongside, so read the blank id from the file rather than hard-coding it.

ONNX

The ONNX graphs are chunk-by-chunk streaming with cache states, so they are not bit-identical to full-utterance PyTorch decoding. Accuracy on held-out real-phone Qur'an:

Runtime	PER
PyTorch, offline full-utterance	8.83%
ONNX fp32, streaming	10.51%
ONNX int8, streaming	10.74%

The 1.7 point difference is the expected cost of limited lookahead relative to full-utterance context. INT8 costs a further 0.23 points while being 3.6x smaller, making it the default choice on device.

Streaming configuration is stored in the ONNX metadata (decode_chunk_len=48, T=61, left_context_len=256,128,64,32,64,128), so sherpa-onnx reads it automatically.

CoreML (Apple Neural Engine)

Two multifunction packages (iOS 18+), identical graph and interface, differing only in weight precision: zipformer_p_arabic_v2.float8.mlpackage (8-bit per-channel weights, fp16 compute, 63 MB, recommended) and zipformer_p_arabic_v2.float16.mlpackage (124 MB). Each contains the encoder as two functions, front and back, sharing one weight blob. It is the same cache-aware streaming computation as the ONNX, and both are device-verified: streaming decode on an iPhone 15 Pro Max (iOS 26) with self-carried state matches the int8 ONNX token for token on every compute-unit configuration. On the float8 package cpuAndNeuralEngine is the fastest configuration outright.

Why two functions. The ANE compiler has a per-program resource budget. This encoder (2,758 compute ops) exceeds it, and the failure mode is silent: ANECCompile() FAILED, then the whole model runs on CPU with no error surfaced (under .all it hard-fails with error -5). Splitting at the narrowest point of the network, where a single 24 KB tensor crosses, gives two programs the compiler accepts. Apple's own ml-stable-diffusion chunks its UNet for the same reason. Earlier monolithic CoreML exports of this model, including the packages previously published here, never actually executed on the ANE for this reason; they are removed.

Three numerical fixes are baked in relative to those earlier exports: the Swoosh overflow guard is a threshold compare instead of an inf compare (the ANE and device GPU saturate fp16 at 65,504 and never produce inf, which silently corrupted activations by up to ~80), the output head is a max-subtracted log-softmax (the ANE computes naive log-sum-exp, which saturates and produced positive log-probs), and the 85 leftover fp32 epsilon parameters that blocked device compilation are gone. Credit to the QuranTracker engine team, whose field report started this investigation.

Interface. The 98 cache tensors are packed by shape into a few blob tensors so each function fits the ANE's input budget. packing_front.json and packing_back.json describe each blob: member states, concat axis, shapes. You do not need to unpack anything at runtime; carry blobs whole.

Per chunk (0.48 s of audio):

Call	Inputs	Outputs
`front`	`x` (1, 61, 80) fbank chunk, blobs `sg_0 .. sg_13`	stream tensor `x_631_cast_fp16` (24, 1, 384), `new_sg_0 .. new_sg_13`
`back`	`x_631_cast_fp16` from front, `state_97` (pre-update, see below), blobs `sg_0 .. sg_11`	`log_probs` (1, 12, 251), `new_sg_0 .. new_sg_11`

State carry is blob to blob: feed each new_sg_k output as next chunk's sg_k input, zero-init on the first call. One subtlety: back takes the raw frame counter state_97 and must receive the value front consumed this chunk, not the updated one, so read it from the front state blob before overwriting (the blob holding it is the one whose manifest lists state_97 as its only member).

let cfg = MLModelConfiguration()
cfg.computeUnits = .cpuAndNeuralEngine
cfg.functionName = "front"
let front = try MLModel(contentsOf: compiledURL, configuration: cfg)
cfg.functionName = "back"
let back = try MLModel(contentsOf: compiledURL, configuration: cfg)
// per chunk: front.prediction -> back.prediction, carry new_sg_k -> sg_k

Measured on iPhone 15 Pro Max (iOS 26), float8 package. Correctness: 14-chunk streaming decode identical to the int8 ONNX on cpuOnly, cpuAndNeuralEngine, cpuAndGPU and all. Dispatch under cpuAndNeuralEngine: back 1,280 of 1,530 ops on the ANE, front 741 of 1,255.

Speed, per 0.48 s chunk:

Configuration	Interactive p50	Sustained (100% duty, thermal nominal)	Realtime multiple
`cpuAndNeuralEngine`	6.8 ms	18.5 ms	70x interactive
`cpuOnly`	7.3 ms	21.6 ms	66x interactive
`cpuAndGPU`	16 ms	not measured	30x

Power, from on-device Power Profiler traces (discharging, battery-health calibrated), net of the screen-on idle baseline:

Package	Configuration	Net power at 100% duty	Energy per chunk	Added draw at realtime streaming
float8	`cpuAndNeuralEngine`	2.26 W	41.7 mJ	~87 mW
float8	`cpuOnly`	2.05 W	44.2 mJ	~92 mW
float16	`cpuAndNeuralEngine`	2.50 W	43 mJ	~90 mW
float16	`cpuOnly`	2.88 W	52 mJ	~109 mW

The float8 rows were captured with the device thermally nominal for both full windows; the float16 rows ran hotter (the cpuOnly window entirely thermally throttled), so compare float16 against float8 directionally rather than to the millijoule. On float8 the ANE configuration draws slightly more instantaneous power because it completes 17% more chunks per second; per unit of work it is 6% more energy-efficient and 14% faster, so it wins both speed and efficiency. Both float8 configurations draw less power than their float16 counterparts, consistent with 8-bit weights halving the weight traffic. Relative to the previously published monolithic CoreML export, the split pair is about 1.9x faster per chunk. Recommended: the float8 package with cpuAndNeuralEngine.

Two integration notes from the field: log_probs arrives with padded strides (1, 12, 251) as [3072, 256, 1], so honour MLMultiArray.strides rather than indexing contiguously, and MLMultiArray(shape:dataType:) returns uninitialised memory, so explicitly zero the state blobs on the first call.

The iOS Simulator does not execute this package correctly (silent zero output on .all); validate on macOS or a physical device.

Per-phoneme confidence, alternatives and timings

Comparing confidence across runtimes

Confidence margins are not automatically transferable between runtimes even when decodes are identical. Greedy emission boundaries can shift by one frame between engines (ONNX, CoreML, PyTorch) on near-tie transition frames, so a margin sampled at a token's first emission frame can read 0.99 on one engine and 0.02 on another while the underlying posteriors agree to the third decimal. Measured on 33 recitations with identical decodes: first-frame margins diverged up to 0.97 between ONNX int8 and CoreML; margins evaluated at each token's peak-probability frame agreed to 0.11 worst case, with every residual gap on tokens that are genuinely uncertain on both engines. decode_with_confidence.py reports this as margin_peak; use it for threshold decisions, and recalibrate on your own corpus if you switch runtimes. Thanks to the QuranTracker team for the field data behind this.

Greedy CTC keeps the argmax and discards the rest of the 251-way posterior. For recitation grading that discarded information is useful, because low model certainty and a genuine mispronunciation are indistinguishable in a plain transcript. decode_with_confidence.py retains it, so every emitted phoneme carries a confidence, its runner-up candidates and a time span.

$ python decode_with_confidence.py clip.wav --topk 2

ءِننننَرَببَهُ۾۾۾بِهِميَومَءِذِللَخَبِۦۦر

         time  ph       conf  margin  alternatives
  1.20- 1.24  ننننَ   1.000   1.000  <blank>:0.00  نَ:0.00
  4.52- 4.60  ءِ      0.790   0.581  <blank>:0.21  ا:0.00
  5.44- 5.48  خَ      1.000   0.999  حَ:0.00  <blank>:0.00
  5.72- 5.80  بِ      0.894   0.789  <blank>:0.10  رِ:0.00

Each entry gives phoneme, confidence (0 to 1), margin (confidence minus runner-up), start and end in seconds, and alternatives. Pass --json out.json for machine-readable output.

CTC posteriors are peaky. On the clip above the mean confidence is 0.983 and no value falls below 0.5, so a fixed confidence < 0.5 gate will rarely fire and high confidence is not evidence of correctness. Two signals carry more information:

margin rather than confidence. A phoneme at 0.79 with <blank> at 0.21 is a contested position; one at 1.000 with all alternatives at 0.00 is not.
The identity of the runner-up. <blank> competing indicates uncertainty that anything was said at that position (onset, pause, clipped sound). A competing phoneme, for example خَ against حَ, indicates a sound was heard but its identity is ambiguous, which is the useful case for pronunciation feedback.

Thresholds are not calibrated probabilities and should be tuned on your own audio. For true calibration, temperature-scale the logits against a labelled set of your own recordings.

Timing resolution is one encoder frame, 40 ms. This separates phonemes reliably but is coarse for grading a precise 2-harakah hold, so treat spans as boundaries rather than fine duration measurements.

Usage

icefall Zipformer2, 80-bin kaldi fbank (povey window) at 16 kHz, CTC greedy decode to phoneme units.

Use torchaudio.compliance.kaldi.fbank, not torchaudio.transforms.MelSpectrogram. Slaney-mel features roughly double the error rate (measured: 11.4% to 22.1% WER on identical audio) without producing any error, and are the most common cause of poor results with this model.

import torch, torchaudio.compliance.kaldi as kaldi

feats = kaldi.fbank(wav, num_mel_bins=80, frame_length=25.0, frame_shift=10.0,
                    sample_frequency=16000, dither=0.0, snip_edges=False,
                    window_type="povey", use_energy=False, low_freq=20.0,
                    high_freq=-400.0, remove_dc_offset=True, preemphasis_coefficient=0.97)

For the 1000 ms look-ahead profile use chunk_size=(24,) and left_context_frames=(256,).

The ONNX files use the standard sherpa-onnx online zipformer2-CTC format (model_type=zipformer2), so they load directly in sherpa-onnx, and in plain onnxruntime by feeding one chunk plus the cache tensors and carrying the returned states forward.

When streaming ends, flush the buffer by passing one final chunk of silence (at least the right-context length, about 1.05 s). Audio still held in the right-context buffer is not emitted otherwise, which causes short final words to be dropped.

Limitations

Phonemes, not text. Converting Arabic phonemes to orthography is a separate problem. Evaluate this model by PER rather than word-level text WER.
Retrieval is a Qur'an-lexicon constraint. The leaderboard WER above uses nearest-ayah matching and does not transfer to general Arabic.
Dialectal orthography is out of scope. The model transcribes dialect faithfully at the phonetic level, but reconstructing fluent dialectal text from it is a separate task.
Broad-Arabic labels are LLM-generated and carry some vowel-length noise.
Emphatic consonants (ط ض ظ ص) degrade on reciters absent from training. Measured on a held-out unseen reciter, ط reaches 24.4% error against 2.6% for ت, while on reciters seen in training ط is 9.3%. The cause is speaker generalisation rather than noise, and reciter diversity is the effective remedy.
The CoreML builds have not been executed on Apple hardware, since coremltools cannot run a model outside macOS. They are conversion-verified and precision-verified only.

License

Shared freely for the sake of Allah under free, non-commercial terms, not Apache-2.0. Access is gated: free apps only, no sales, no paid subscriptions or paywalls, no ad revenue, and no monetization of any kind. These terms pass on to anyone you share it with. See LICENSE.

Downloads last month: 30

Muno459
/

zipformer_p-arabic-v2