Instructions to use Menlo/Jan-nano-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Menlo/Jan-nano-gguf with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Menlo/Jan-nano-gguf",
	filename="jan-nano-4b-Q3_K_L.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Menlo/Jan-nano-gguf with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Menlo/Jan-nano-gguf:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Menlo/Jan-nano-gguf:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Menlo/Jan-nano-gguf:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Menlo/Jan-nano-gguf:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Menlo/Jan-nano-gguf:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Menlo/Jan-nano-gguf:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Menlo/Jan-nano-gguf:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Menlo/Jan-nano-gguf:Q4_K_M

Use Docker

docker model run hf.co/Menlo/Jan-nano-gguf:Q4_K_M

LM Studio
Jan

vLLM

How to use Menlo/Jan-nano-gguf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Menlo/Jan-nano-gguf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Menlo/Jan-nano-gguf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Menlo/Jan-nano-gguf:Q4_K_M

Ollama
How to use Menlo/Jan-nano-gguf with Ollama:
```
ollama run hf.co/Menlo/Jan-nano-gguf:Q4_K_M
```

Unsloth Studio

How to use Menlo/Jan-nano-gguf with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Menlo/Jan-nano-gguf to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Menlo/Jan-nano-gguf to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Menlo/Jan-nano-gguf to start chatting

How to use Menlo/Jan-nano-gguf with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf Menlo/Jan-nano-gguf:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Menlo/Jan-nano-gguf:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use Menlo/Jan-nano-gguf with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf Menlo/Jan-nano-gguf:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Menlo/Jan-nano-gguf:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use Menlo/Jan-nano-gguf with Docker Model Runner:
```
docker model run hf.co/Menlo/Jan-nano-gguf:Q4_K_M
```

Lemonade

How to use Menlo/Jan-nano-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Menlo/Jan-nano-gguf:Q4_K_M

Run and chat with the model

lemonade run user.Jan-nano-gguf-Q4_K_M

List all available models

lemonade list

Failed to parse Jinja template: Parser Error: Expected closing statement token. OpenSquareBracket !== CloseStatement.

by ljupco - opened Jun 14, 2025

Discussion

ljupco

Jun 14, 2025

Trying to run it on MacOS LMStudio. I put the chat template from

https://huggingface.co/Menlo/Jan-nano?chat_template=default&format=true

into LMStudio's "Model default parameters" - "Prompt" - "Template (Jinja)" box, but I'm getting

Failed to parse Jinja template: Parser Error: Expected closing statement token. OpenSquareBracket !== CloseStatement.

Help! :-)

alandao

Menlo Research org Jun 15, 2025

Hi you can use Qwen3 template from other lmstudio compatible model but remember to disable "thinking" and add this system prompt when using

In this environment you have access to a set of tools you can use to answer the user's question. You can use one tool per message, and will receive the result of that tool use in the user's response. You use tools step-by-step to accomplish a given task, with each tool use informed by the result of the previous tool use.

Tool Use Rules
Here are the rules you should always follow to solve your task:
1. Always use the right arguments for the tools. Never use variable names as the action arguments, use the value instead.
2. Call a tool only when needed: do not call the search agent if you do not need information, try to solve the task yourself.
3. If no tool call is needed, just answer the question directly.
4. Never re-do a tool call that you previously did with the exact same parameters.
5. For tool use, MARK SURE use XML tag format as shown in the examples above. Do not use any other format.

In the meantime we will try to see if we can fix the gguf

Enjoy

ljupco

Jun 15, 2025

Thanks! I can confirm the model is working great in Jan. (the App) Thanks for your help.

limcheekin

Jun 16, 2025

Thanks! I can confirm the model is working great in Jan. (the App) Thanks for your help.

Great! May I know which version of Jan are you using?

ljupco

Jun 17, 2025

Certainly! I'm using

Jan Version v0.5.17

on an oldlish MBP M2 with 96GB of RAM.

Thanks for your help and all your work on this. Really enjoying messing about with the local models. :-)

ThomasBbm

Jun 27, 2025

Forgive the noob but how does one " disable "thinking" " for a model?

jan-hq

Menlo Research org Jun 27, 2025

only Qwen3 family model has ability to disable thinking by passing disable_thinking=True to tokenizer.

Basher17

Jul 2, 2025

Hi, I just found this model and it looks really promising. Only one thing: could you please paste here the Qwen3 chat template with thinking disabled?
I'm kind of a noob, wasn't able to figure out how to do it myself. Thanks!

alandao

Menlo Research org Jul 2, 2025

Hi, I just found this model and it looks really promising. Only one thing: could you please paste here the Qwen3 chat template with thinking disabled?
I'm kind of a noob, wasn't able to figure out how to do it myself. Thanks!

Try this

https://huggingface.co/Menlo/Jan-nano-128k-gguf/discussions/1#6862fe2375cb85f79b28d69c

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment