Typhoon OCR V2 0.8B GGUF

GGUF files of Typhoon OCR V2 0.8B for llama.cpp and tools built on it.

Typhoon OCR V2 0.8B is a document parser for Thai and English. Given an image of a page, it returns the page's layout elements in reading order, each with a category, a bounding box and its text. Tables come out as HTML, formulas as LaTeX and other text as Markdown. It also answers questions about a document and extracts key information as JSON. The main model card has the training details, full benchmark results, examples and limitations.

Use of this model is subject to the OpenTyphoon Terms and Conditions.

Files and accuracy

File Precision Size MDPBench โ†‘ MDPBench Thai โ†‘
Original weights (bf16, vLLM) โ€“ โ€“ 77.4 77.2
typhoon-ocr2-0.8b-f16.gguf 16-bit float, no quantization 1.52 GB 77 76.9
typhoon-ocr2-0.8b-Q8_0.gguf 8-bit 812 MB 77.4 78
typhoon-ocr2-0.8b-Q4_K_M.gguf 4-bit (k-quant, medium) 529 MB 75.8 76.7
mmproj-typhoon-ocr2-0.8b-f16.gguf image encoder (f16), needed with every file above 205 MB

MDPBench has 320 Thai and English pages, both digital and photographed. We ran every file through llama-server (llama.cpp build a691b23, CUDA, NVIDIA H100) with the prompt below, greedy decoding, thinking off and an image budget of 16,384 tokens (16.7 MP), the same settings as the vLLM numbers in the first row. We recommend Q8_0: it scored as well as f16 and the original weights (77.4) at about half the size of f16. Q4_K_M is about 1.5 points lower overall. Differences of a few tenths of a point between files are within run-to-run variation.

Usage with llama.cpp

-hf downloads the model file and the matching mmproj image encoder:

llama-server -hf typhoon-ai/typhoon-ocr2-0.8b-GGUF:Q8_0 \
  --jinja --reasoning off \
  -c 40960 --image-max-tokens 16384 \
  -ngl 999 --port 8080

With local files, pass them directly: -m typhoon-ocr2-0.8b-Q8_0.gguf --mmproj mmproj-typhoon-ocr2-0.8b-f16.gguf.

  • --image-max-tokens 16384 lets a page use up to 16.7 MP (one image token covers 32ร—32 pixels). Without it, llama.cpp scales large pages down further, and small text gets harder to read.
  • --reasoning off keeps the chat template from opening a thinking block. The model was trained without one.
  • -c 40960 leaves room for a 16,384-token image and a 16,384-token answer. To serve several requests at once, add -np N and multiply -c by N.

Then call the OpenAI-compatible endpoint, with PROMPT and the helpers from the next section:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="EMPTY")

image_url = to_data_url(load_image("page.jpg"))

resp = client.chat.completions.create(
    model="typhoon-ocr2",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": image_url}},
        {"type": "text", "text": PROMPT},
    ]}],
    temperature=0.0,
    max_tokens=16384,
)
elements = parse_elements(resp.choices[0].message.content)
print(to_markdown(elements))

Prompt and output

The model was trained on a fixed set of prompts. For full-page parsing, use this prompt exactly as written:

PROMPT = """- Extract layout information from the provided PDF image.
- For each layout element, output its bbox, category, and the text content within the bbox.
- Bbox format: [x1, y1, x2, y2].
- Allowed layout categories: ['header', 'title', 'text', 'figure', 'table', 'formula', 'figure_caption', 'table_caption', 'formula_caption', 'figure_footnote', 'table_footnote', 'page_footnote', 'footer'].
- Text extraction and formatting:
  1) For 'figure', the text field must be an empty string.
  2) For 'formula', format text as LaTeX.
  3) For 'table', format text as HTML.
  4) For all other categories (e.g., text, title), format text as Markdown.
- The output text must be exactly the original text from the image, with no translation or rewriting.
- Sort all layout elements in human reading order.
- Final output must be a single JSON object."""

Before sending a page, enlarge it so that its long side is at least 1,600 pixels, keeping the aspect ratio. This is required: the model reads small images poorly, Thai text most of all. load_image below does it:

import base64, io
from PIL import Image

def upscale_min_long(image: Image.Image, min_long: int = 1600) -> Image.Image:
    """Enlarge the image so that its long side is at least `min_long` pixels, keeping the
    aspect ratio. Images that are already large enough are returned unchanged."""
    scale = min_long / max(image.size)
    if scale <= 1:
        return image
    return image.resize((round(image.width * scale), round(image.height * scale)), Image.LANCZOS)

def load_image(path: str) -> Image.Image:
    return upscale_min_long(Image.open(path).convert("RGB"))

def to_data_url(image: Image.Image) -> str:
    buf = io.BytesIO()
    image.save(buf, format="PNG")
    return "data:image/png;base64," + base64.b64encode(buf.getvalue()).decode()

The model returns a JSON list of elements inside a json code block. Bounding boxes use a 0โ€“1000 grid relative to the page width and height, so they map back onto the original image. These helpers parse the output and turn it into Markdown:

import json, re

def parse_elements(raw: str) -> list[dict]:
    """Parse the model output: a JSON list of elements inside a ```json fence."""
    m = re.search(r"```json\n(.*?)(?:\n```|$)", raw, re.DOTALL)
    text = (m.group(1) if m else raw).strip()
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        # Output stopped at the token limit: drop the last, incomplete element.
        cut = text.rfind('{"bbox":')
        return json.loads(text[:cut].rstrip().rstrip(",") + "]")

def to_markdown(elements: list[dict], keep_header_footer: bool = False) -> str:
    skip = set() if keep_header_footer else {"header", "footer", "page_footnote"}
    return "\n\n".join(e["text"] for e in elements if e.get("text") and e["category"] not in skip)

For questions about a document, send the question as the text prompt instead. For key-information extraction and the other tasks, see the main model card.

Limitations

  • Quantization costs some accuracy; compare the scores above before picking a smaller file.
  • We tested llama.cpp on Linux with CUDA only. CPU, Metal and Vulkan builds were not tested, and their outputs can differ slightly.
  • The model can fall into repetition loops on dense multi-column pages. parse_elements above keeps the complete elements when that happens.
  • All limitations of the original model apply.

Support

Downloads last month
614
GGUF
Model size
0.8B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

4-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for typhoon-ai/typhoon-ocr2-0.8b-GGUF

Quantized
(5)
this model