GPT-OSS-Code-Reasoning-20B-GGUF

This is the GGUF quantized version of the GPT-OSS-Code-Reasoning-20B model, optimized for efficient inference with reduced memory requirements.

Overview

• Base model: openai/gpt-oss-20b
• Objective: Supervised fine-tuning for competitive programming and algorithmic reasoning
• Format: GGUF (optimized for llama.cpp and compatible inference engines)

Model Variants

This GGUF model is available in multiple quantization levels to suit different hardware requirements:

Quantization	Size	Memory Usage	Quality
Q3_K_M	12.9 GB	~13 GB	Average
Q4_K_M	15.8 GB	~16 GB	Good
Q5_K_M	16.9 GB	~17 GB	Better
Q8_0	22.3 GB	~23 GB	Best

Intended Use

• Intended: Generating Python/C++ solutions and reasoning for competitive programming tasks
• Out of scope: Safety-critical applications. May hallucinate or produce incorrect/inefficient code

Quick Start

Using llama.cpp

# Download the model
wget https://huggingface.co/GetSoloTech/GPT-OSS-Code-Reasoning-20B-GGUF/resolve/main/gpt-oss-code-reasoning-20b.Q4_K_M.gguf

# Run inference
./llama.cpp -m gpt-oss-code-reasoning-20b.Q4_K_M.gguf -n 512 --repeat_penalty 1.1

Using Python with llama-cpp-python

from llama_cpp import Llama

# Load the model
llm = Llama(
    model_path="./gpt-oss-code-reasoning-20b.Q4_K_M.gguf",
    n_ctx=4096,
    n_threads=8
)

# Example problem
problem_text = """
You are given an array of integers nums and an integer target. 
Return indices of the two numbers such that they add up to target.
"""

# Create the prompt
prompt = f"""<|im_start|>system
You are an expert competitive programmer. Read the problem and produce a correct, efficient solution. Include reasoning if helpful.
<|im_end|>
<|im_start|>user
{problem_text}
<|im_end|>
<|im_start|>assistant
"""

# Generate response
output = llm(
    prompt,
    max_tokens=768,
    temperature=0.3,
    top_p=0.9,
    repeat_penalty=1.1,
    stop=["<|im_end|>"]
)

print(output['choices'][0]['text'])

Using Ollama

# Create a Modelfile
cat > Modelfile << EOF
FROM ./gpt-oss-code-reasoning-20b.Q4_K_M.gguf
TEMPLATE """<|im_start|>system
{{ .System }}
<|im_end|>
<|im_start|>user
{{ .Prompt }}
<|im_end|>
<|im_start|>assistant
"""
PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1
EOF

# Create and run the model
ollama create code-reasoning -f Modelfile
ollama run code-reasoning "Solve this competitive programming problem: [your problem here]"

Prompt Format

This model was trained in a chat format. Recommended structure:

messages = [
    {"role": "system", "content": "You are an expert competitive programmer. Read the problem and produce a correct, efficient solution. Include reasoning if helpful."},
    {"role": "user", "content": problem_text},
]

For GGUF models, use the following format:

<|im_start|>system
You are an expert competitive programmer. Read the problem and produce a correct, efficient solution. Include reasoning if helpful.
<|im_end|>
<|im_start|>user
{problem_text}
<|im_end|>
<|im_start|>assistant

Generation Tips

• Reasoning style: Lower temperature (0.2–0.5) for clearer step-by-step reasoning
• Length: Use max_tokens 512–1024 for full solutions; shorter for hints
• Stop tokens: The model uses <|im_end|> as a stop token
• Memory optimization: Choose the appropriate quantization level based on your hardware

Hardware Requirements

Quantization	Minimum RAM	Recommended RAM	GPU VRAM
Q3_K_M	8 GB	16 GB	8 GB
Q4_K_M	12 GB	24 GB	12 GB
Q5_K_M	16 GB	32 GB	16 GB
Q8_0	24 GB	48 GB	24 GB

Performance Notes

• Speed: GGUF models are optimized for fast inference
• Memory: Significantly reduced memory footprint compared to the original model
• Quality: Minimal quality loss with appropriate quantization levels
• Compatibility: Works with llama.cpp, llama-cpp-python, Ollama, and other GGUF-compatible engines

Acknowledgements

• Original model: GetSoloTech/GPT-OSS-Code-Reasoning-20B
• Base model: openai/gpt-oss-20b
• Dataset: nvidia/OpenCodeReasoning-2
• Upstream benchmarks: TACO, APPS, DeepMind CodeContests, open-r1/codeforces