Maba-101M: Pretrained Checkpoint and Benchmark Evaluation

Official weights and benchmark evaluations for the 101M Maba v1.1 model (101,177,984 total parameters, 96,327,040 core computation parameters).

Architecture Specification and C++ Source Code For the complete architectural specification, layer mathematics, C++ native inference engine, and test suites, see the core architecture repository: AndrewThompson1233/maba-v1-architecture

---

---

4-Way Model Showdown (~101M Parameters)

All 4 models were evaluated under an equalized parameter budget (~101M parameters) trained on the exact same 16,000,000 tokens of TinyStories and evaluated under identical conditions:

Table 1: Standardized Benchmark Results (3,000 Total Tasks)

Architecture	ARC-Easy (250)	HellaSwag (250)	Story-Cloze (250)	Val Loss (500 seq)	Val PPL (500 seq)	Rank
Maba v1.1 (101M)	26.80%	24.00%	25.20%	5.8787	357.34	1
MiniCPM5 (101M)	25.60%	23.60%	21.60%	5.9476	382.84	2
Qwen 3.8 Flash Next (101M)	23.60%	25.60%	25.20%	6.1351	461.80	3
Qwen 3.8 (101M)	25.20%	23.60%	25.60%	6.1538	470.51	4
Random Guessing Baseline	25.00%	25.00%	25.00%	N/A	N/A	Baseline

Table 2: Architecture Specifications (~101M Parameter Budget)

Parameter	Maba v1.1	Qwen 3.8	Qwen 3.8 Flash Next	MiniCPM5
Exact Parameters	101,177,984 (101.18M)	101,152,384 (101.15M)	101,126,824 (101.13M)	100,403,392 (100.40M)
Computation Core	96,327,040 (95.21%)	75,864,064 (75.00%)	75,838,504 (74.99%)	100,403,392 (100.0%)
Layer Composition	75% GDN-2 + 25% GQA	75% GDN + 25% GQA	75% GDN + 25% QSA	100% GQA
Physical Blocks	20 blocks	20 blocks	20 blocks	28 blocks
Effective Layers	40 layers (2-pass recycling)	20 layers (1 pass)	20 layers (1 pass)	28 layers (1 pass)
Attention Mechanism	GQA (d_head=64, kv=2)	GQA (d_head=64, kv=2)	QSA (Micro-block Sparse)	GQA (d_head=48, kv=2)
Residual Type	Gated Residual	Standard Residual	Dual-Gated Residual	Standard Residual
Speculative Head	MTP (k=2 built-in)	MTP (k=2 built-in)	MTP (k=2 built-in)	None

Table 3: Memory and Runtime Throughput (v1.1 Benchmarks)

Metric	Maba v1.1	Qwen 3.8	Qwen 3.8 Flash Next	MiniCPM5
KV Cache (4k Physical)	10,240 KB (10.0 MB)	10,240 KB (10.0 MB)	2,560 KB (2.5 MB)	43,008 KB (42.0 MB)
KV Cache (4k Runtime)	10,240 KB (10.0 MB)	10,240 KB (10.0 MB)	2,560 KB (2.5 MB)	43,008 KB (42.0 MB)
KV Cache Reduction	-76.2%	-76.2%	-94.0%	0.0% (Baseline)
Inference Throughput	394.2 tok/s	171.4 tok/s	157.5 tok/s	278.2 tok/s
Training Speed (4x L4)	38,400 tok/s	~1,360 tok/s	~1,290 tok/s	~9,455 tok/s
Reasoning Margin	+0.2237 (Best)	+0.1809	+0.1618	+0.1754

---

Model Details

• Architecture: Maba v1.1 (Hybrid GDN-2 Recurrence + GQA + MTP)
• Total Parameters: 101,177,984
• Core Computation Parameters: 96,327,040
• Hidden Size: 640
• Physical Layers: 20 (15 GDN-2 + 5 GQA)
• Effective Layers: 40 (2 passes)
• Query Heads: 10 (d_head = 64)
• KV Heads: 2
• FFN Dimension: 1,728 (SwiGLU)
• Vocabulary Size: 32,768 (Embedding rank 128)
• Context Window: 4,096 tokens
• Precision: bfloat16 / float32

---

Usage

Loading Pretrained Weights

import torch
from maba.model import Model
from maba.config import Config
from maba.tokenizer import Tokenizer
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

# Initialize model
cfg = Config.from_preset("100M")
model = Model(cfg)

# Load safetensors weights
weights_path = hf_hub_download(repo_id="AndrewThompson1233/maba-101m", filename="model.safetensors")
state_dict = load_file(weights_path)
model.load_state_dict(state_dict)
model.eval()

tok = Tokenizer()
prompt = "Once upon a time in a quiet village"
input_ids = torch.tensor([tok.encode(prompt, add_bos=True)])

with torch.no_grad():
    output_ids = model.generate(input_ids, max_new_tokens=40, temperature=0.7)

print(tok.decode(output_ids[0].tolist()))

High-Speed Speculative Generation (k=2)

from maba.generate import spec_gen

output_text, acceptance_rate, steps = spec_gen(
    model,
    tok,
    prompt="A young inventor built a clockwork bird",
    max_new_tokens=60
)
print(output_text)