--- license: apache-2.0 base_model: - HuggingFaceTB/SmolLM2-360M-Instruct base_model_relation: quantized library_name: executorch pipeline_tag: text-generation tags: - android - executorch - mtk - on-device - openweights - vulkan - xnnpack --- # SmolLM2-360M-Instruct for ExecuTorch ExecuTorch exports of [HuggingFaceTB/SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct) (revision `a10cc1512eab`) for on-device inference with the [openweights](https://github.com/alpharomercoma/openweights) Android app or any ExecuTorch 1.4.0 runtime. ## Files Windows in this repo: XNNPACK (CPU) at 2k to 32k; Vulkan (GPU) at 2k to 32k; MediaTek NeuroPilot MT6991 (Dimensity 9400) at 2k to 16k. The window is fixed inside the file: the runtime allocates the whole KV cache at load, so pick the largest window the device can hold (`fits_phone_budget` in each folder's `config.json` is the estimate against a 5 GB budget). | Backend | Target | File | Window | Size | Smoke test | |---|---|---|---|---|---| | XNNPACK (CPU) | any arm64 | [`xnnpack/SmolLM2-360M-Instruct-fp32-2k.pte`](xnnpack/SmolLM2-360M-Instruct-fp32-2k.pte) | 2,048 tokens | 1.50 GB | passed ("Paris") | | XNNPACK (CPU) | any arm64 | [`xnnpack/SmolLM2-360M-Instruct-fp32-4k.pte`](xnnpack/SmolLM2-360M-Instruct-fp32-4k.pte) | 4,096 tokens | 1.50 GB | passed ("Paris") | | XNNPACK (CPU) | any arm64 | [`xnnpack/SmolLM2-360M-Instruct-fp32-8k.pte`](xnnpack/SmolLM2-360M-Instruct-fp32-8k.pte) | 8,192 tokens | 1.50 GB | passed ("Paris") | | XNNPACK (CPU) | any arm64 | [`xnnpack/SmolLM2-360M-Instruct-fp32-16k.pte`](xnnpack/SmolLM2-360M-Instruct-fp32-16k.pte) | 16,384 tokens | 1.50 GB | passed ("Paris") | | XNNPACK (CPU) | any arm64 | [`xnnpack/SmolLM2-360M-Instruct-fp32-32k.pte`](xnnpack/SmolLM2-360M-Instruct-fp32-32k.pte) | 32,768 tokens | 1.51 GB | passed ("Paris") | | Vulkan (GPU) | any arm64 | [`vulkan/SmolLM2-360M-Instruct-vulkan-8da4w-2k.pte`](vulkan/SmolLM2-360M-Instruct-vulkan-8da4w-2k.pte) | 2,048 tokens | 0.32 GB | structure checked (no host NPU runtime) | | Vulkan (GPU) | any arm64 | [`vulkan/SmolLM2-360M-Instruct-vulkan-8da4w-4k.pte`](vulkan/SmolLM2-360M-Instruct-vulkan-8da4w-4k.pte) | 4,096 tokens | 0.33 GB | structure checked (no host NPU runtime) | | Vulkan (GPU) | any arm64 | [`vulkan/SmolLM2-360M-Instruct-vulkan-8da4w-8k.pte`](vulkan/SmolLM2-360M-Instruct-vulkan-8da4w-8k.pte) | 8,192 tokens | 0.33 GB | structure checked (no host NPU runtime) | | Vulkan (GPU) | any arm64 | [`vulkan/SmolLM2-360M-Instruct-vulkan-8da4w-16k.pte`](vulkan/SmolLM2-360M-Instruct-vulkan-8da4w-16k.pte) | 16,384 tokens | 0.35 GB | structure checked (no host NPU runtime) | | Vulkan (GPU) | any arm64 | [`vulkan/SmolLM2-360M-Instruct-vulkan-8da4w-32k.pte`](vulkan/SmolLM2-360M-Instruct-vulkan-8da4w-32k.pte) | 32,768 tokens | 0.38 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-2k-chunk1of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-2k-chunk1of4.pte) | 2,048 tokens | 0.09 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-2k-chunk2of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-2k-chunk2of4.pte) | 2,048 tokens | 0.09 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-2k-chunk3of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-2k-chunk3of4.pte) | 2,048 tokens | 0.09 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-2k-chunk4of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-2k-chunk4of4.pte) | 2,048 tokens | 0.14 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-4k-chunk1of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-4k-chunk1of4.pte) | 4,096 tokens | 0.09 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-4k-chunk2of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-4k-chunk2of4.pte) | 4,096 tokens | 0.09 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-4k-chunk3of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-4k-chunk3of4.pte) | 4,096 tokens | 0.09 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-4k-chunk4of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-4k-chunk4of4.pte) | 4,096 tokens | 0.14 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-8k-chunk1of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-8k-chunk1of4.pte) | 8,192 tokens | 0.10 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-8k-chunk2of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-8k-chunk2of4.pte) | 8,192 tokens | 0.10 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-8k-chunk3of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-8k-chunk3of4.pte) | 8,192 tokens | 0.10 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-8k-chunk4of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-8k-chunk4of4.pte) | 8,192 tokens | 0.14 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-16k-chunk1of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-16k-chunk1of4.pte) | 16,384 tokens | 0.10 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-16k-chunk2of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-16k-chunk2of4.pte) | 16,384 tokens | 0.10 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-16k-chunk3of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-16k-chunk3of4.pte) | 16,384 tokens | 0.10 GB | structure checked (no host NPU runtime) | | MediaTek NeuroPilot | MT6991 (Dimensity 9400) | [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-16k-chunk4of4.pte`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-a16w8-16k-chunk4of4.pte) | 16,384 tokens | 0.14 GB | structure checked (no host NPU runtime) | MediaTek folders also hold the token embedding table the NeuroPilot runner reads from disk, shared by every window: [`mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-embedding-fp32.bin`](mtk/mt6991/SmolLM2-360M-Instruct-neuropilot-embedding-fp32.bin). Tokenizer: [`tokenizer.json`](tokenizer.json), copied unchanged from the source repo. Each backend folder has a `config.json` listing every window as a variant with the metadata the `.pte` reports, and an `export-report-.json` per file with the full export record. ## Memory - XNNPACK (CPU) at 2,048 tokens: the KV cache costs 81,920 bytes per token (fp32), 167,772,160 bytes for the whole window, allocated in full when the model loads. - XNNPACK (CPU) at 4,096 tokens: the KV cache costs 81,920 bytes per token (fp32), 335,544,320 bytes for the whole window, allocated in full when the model loads. - XNNPACK (CPU) at 8,192 tokens: the KV cache costs 81,920 bytes per token (fp32), 671,088,640 bytes for the whole window, allocated in full when the model loads. - XNNPACK (CPU) at 16,384 tokens: the KV cache costs 81,920 bytes per token (fp32), 1,342,177,280 bytes for the whole window, allocated in full when the model loads. - XNNPACK (CPU) at 32,768 tokens: the KV cache costs 81,920 bytes per token (fp32), 2,684,354,560 bytes for the whole window, allocated in full when the model loads. - Vulkan (GPU) at 2,048 tokens: the KV cache costs 81,920 bytes per token (fp32), 167,772,160 bytes for the whole window, allocated in full when the model loads. - Vulkan (GPU) at 4,096 tokens: the KV cache costs 81,920 bytes per token (fp32), 335,544,320 bytes for the whole window, allocated in full when the model loads. - Vulkan (GPU) at 8,192 tokens: the KV cache costs 81,920 bytes per token (fp32), 671,088,640 bytes for the whole window, allocated in full when the model loads. - Vulkan (GPU) at 16,384 tokens: the KV cache costs 81,920 bytes per token (fp32), 1,342,177,280 bytes for the whole window, allocated in full when the model loads. - Vulkan (GPU) at 32,768 tokens: the KV cache costs 81,920 bytes per token (fp32), 2,684,354,560 bytes for the whole window, allocated in full when the model loads. ## How it was made - XNNPACK (CPU) any arm64: ExecuTorch 1.4.0 `export_llm`: fp32 weights and activations in every linear (not quantized), int8 per-channel embeddings, XNNPACK with extended ops, prefill chunk 2048, fp32 KV cache. Built by [run 1](https://github.com/ExperimentalMachines/execupack/actions/runs/37160993875). - Vulkan (GPU) any arm64: ExecuTorch 1.4.0 `export_llm`: 8-bit dynamic activations and 4-bit weights in groups of 32, int8 per-channel embeddings, the Vulkan delegate, prefill chunk 2048, fp32 KV cache. Built by [run 1](https://github.com/ExperimentalMachines/execupack/actions/runs/37157383005). - MediaTek NeuroPilot MT6991 (Dimensity 9400): ExecuTorch 1.4.0 MediaTek LLM export (`examples/mediatek`, `llama.py`): A16W8 (16-bit activations, 8-bit weights) calibrated on MediaTek's `alpaca.txt` prompts (9 of 9) in the qwen.json chat template, cut into 4 chunks, with a 128-token prompt graph and a one-token generation graph over a 2048-token cache, compiled with MediaTek NeuroPilot Express SDK (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) for MT6991 (Dimensity 9400). Built by [run 1](https://github.com/ExperimentalMachines/execupack/actions/runs/37158194354). - MediaTek NeuroPilot MT6991 (Dimensity 9400): ExecuTorch 1.4.0 MediaTek LLM export (`examples/mediatek`, `llama.py`): A16W8 (16-bit activations, 8-bit weights) calibrated on MediaTek's `alpaca.txt` prompts (9 of 9) in the qwen.json chat template, cut into 4 chunks, with a 128-token prompt graph and a one-token generation graph over a 4096-token cache, compiled with MediaTek NeuroPilot Express SDK (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) for MT6991 (Dimensity 9400). Built by [run 1](https://github.com/ExperimentalMachines/execupack/actions/runs/37158194354). - MediaTek NeuroPilot MT6991 (Dimensity 9400): ExecuTorch 1.4.0 MediaTek LLM export (`examples/mediatek`, `llama.py`): A16W8 (16-bit activations, 8-bit weights) calibrated on MediaTek's `alpaca.txt` prompts (9 of 9) in the qwen.json chat template, cut into 4 chunks, with a 128-token prompt graph and a one-token generation graph over a 8192-token cache, compiled with MediaTek NeuroPilot Express SDK (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) for MT6991 (Dimensity 9400). Built by [run 1](https://github.com/ExperimentalMachines/execupack/actions/runs/37158194354). - MediaTek NeuroPilot MT6991 (Dimensity 9400): ExecuTorch 1.4.0 MediaTek LLM export (`examples/mediatek`, `llama.py`): A16W8 (16-bit activations, 8-bit weights) calibrated on MediaTek's `alpaca.txt` prompts (9 of 9) in the qwen.json chat template, cut into 4 chunks, with a 128-token prompt graph and a one-token generation graph over a 16384-token cache, compiled with MediaTek NeuroPilot Express SDK (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) for MT6991 (Dimensity 9400). Built by [run 1](https://github.com/ExperimentalMachines/execupack/actions/runs/37158194354). ## License A quantized derivative of [HuggingFaceTB/SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct), distributed under the same terms (apache-2.0). The `mtk/` folders hold model binaries compiled with the MediaTek NeuroPilot Express SDK 8.0.8-build20250925 (mtk_converter 8.13.0+public, mtk_neuron 8.2.23) from MediaTek Inc., used under MediaTek's license terms for that SDK. No MediaTek SDK or runtime library is included. They run on MediaTek's LLM runner from ExecuTorch (`examples/mediatek/executor_runner`) with the device's NeuroPilot runtime; the settings it needs are in each folder's `config.json`, under each variant's `runner`.