Hob-forge/BAR-5x7B-GGUF

BAR-5x7B — GGUF (first-of-its-kind FlexOlmo conversion)

This is the first GGUF conversion of allenai/BAR-5x7B, the largest member of AllenAI's BAR-family Mixture-of-Experts models released on 2026-04-19 based on the new FlexOlmo architecture.

5 experts × 7B → ~33B total parameters with top-k routing.

⚠ Requires a FlexOlmo-enabled llama.cpp

Upstream llama.cpp does not support the FlexOlmo architecture: the support PR ggml-org/llama.cpp#22540 was closed unmerged on 2026-04-30. Use hob-llama.cpp, our llama.cpp build that adds FlexOlmo on a current upstream base:

• Prebuilt binaries: release hob-b11434: Linux (CPU x64/arm64, CUDA 12.8, ROCm, Vulkan), Windows (CPU x64/arm64, CUDA 12.4/13.4, ROCm, Vulkan) and macOS (Intel x64), with SHA256SUMS.txt.
• Or build from source:

git clone https://github.com/Seraphiel102/llama.cpp.git
cd llama.cpp
git checkout hob-b11434
cmake -B build -DGGML_CUDA=OFF        # or -DGGML_CUDA=ON etc. for your GPU
cmake --build build -j --target llama-cli llama-completion llama-server

What FlexOlmo is

Per transformers.models.flex_olmo, FlexOlmoDecoderLayer is Olmo2's hybrid post-norm decoder layer with the dense FFN swapped for OlmoE-style top-k MoE routing. Specifically:

• Attention with q_norm and k_norm (Olmo2-style)
• post_attention_layernorm and post_feedforward_layernorm (post-norm pattern, no input_layernorm)
• Top-k MoE FFN with softmax routing (OlmoE-style)
• No sliding-window attention

Files

Quant	Size	Notes
BAR-5x7B.Q4_K_M.gguf	14 GB	recommended, fits 16GB VRAM at small context
(more quants pending)

Usage

./build/bin/llama-completion \
  -m BAR-5x7B.Q4_K_M.gguf \
  -p "The 5 experts in BAR-5x7B are " \
  -n 100

Validation

The Q4_K_M conversion was validated against the April FlexOlmo build using a basic arithmetic prompt and produces correct, coherent output. BAR-5x7B itself has not yet been re-run on b11434; the FlexOlmo graph there was verified equivalent to the April build (a tiny random FlexOlmo model, plus BAR-2x7B Q4_K_M with byte-identical output).

Credit

• Model: AllenAI — allenai/BAR-5x7B
• FlexOlmo support in llama.cpp: hob-llama.cpp by @Seraphiel102 / Nyx (upstream PR #22540 was closed)
• Conversion: llama.cpp + our FlexOlmo convert_hf_to_gguf.py patch