Instructions to use lightblue/qarasu-14B-chat-plus-unleashed with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use lightblue/qarasu-14B-chat-plus-unleashed with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="lightblue/qarasu-14B-chat-plus-unleashed", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("lightblue/qarasu-14B-chat-plus-unleashed", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use lightblue/qarasu-14B-chat-plus-unleashed with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "lightblue/qarasu-14B-chat-plus-unleashed"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lightblue/qarasu-14B-chat-plus-unleashed",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/lightblue/qarasu-14B-chat-plus-unleashed

SGLang

How to use lightblue/qarasu-14B-chat-plus-unleashed with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "lightblue/qarasu-14B-chat-plus-unleashed" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lightblue/qarasu-14B-chat-plus-unleashed",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "lightblue/qarasu-14B-chat-plus-unleashed" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lightblue/qarasu-14B-chat-plus-unleashed",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use lightblue/qarasu-14B-chat-plus-unleashed with Docker Model Runner:
```
docker model run hf.co/lightblue/qarasu-14B-chat-plus-unleashed
```

qarasu-14B-chat-plus-unleashed / README.md

shun1taniguchi

Update README.md

bb6f42f over 2 years ago

preview code

raw

history blame contribute delete

3.77 kB

metadata

license: other
license_name: tongyi-qianwen-license-agreement
license_link: >-
  https://github.com/QwenLM/Qwen/blob/main/Tongyi%20Qianwen%20LICENSE%20AGREEMENT
datasets:
  - OpenAssistant/oasst1
  - zetavg/ShareGPT-Processed
  - augmxnt/ultra-orca-boros-en-ja-v1
language:
  - ja
  - en

drawing

Qwen/Qwen-14B-Chat + Karasu's finetuning datasets

Demo　・　モデルのデモ

Model demo ・モデルのデモ

Blog post・説明の記事

Evaluation

In our internal evaluations, we find the Qarasu model to have particularly high performance on the MTーBench benchmark. We are currently awaiting external evaluations.

How to use

Hugggingface

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("lightblue/qarasu-14B-chat-plus-unleashed", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("lightblue/qarasu-14B-chat-plus-unleashed", torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

messages = [{"role": "system", "content": "あなたはAIアシスタントです。"}]
messages.append({"role": "user", "content": "イギリスの首相は誰ですか？"})

prompt = tokenizer.apply_chat_template(conversation=messages, add_generation_prompt=True, tokenize=False)

pipe(prompt, max_new_tokens=100, do_sample=False, temperature=0.0, return_full_text=False)

VLLM

from vllm import LLM, SamplingParams

sampling_params = SamplingParams(temperature=0.0, max_tokens=100)
llm = LLM(model="lightblue/qarasu-14B-chat-plus-unleashed", trust_remote_code=True)

messages = [{"role": "system", "content": "あなたはAIアシスタントです。"}]
messages.append({"role": "user", "content": "イギリスの首相は誰ですか？"})
prompt = llm.llm_engine.tokenizer.apply_chat_template(conversation=messages, add_generation_prompt=True, tokenize=False)
prompts = [prompt]

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Base checkpoint

Qwen/Qwen-14B-Chat

Training datasets (total ~7B)

The same as the 'plus' checkpoint, but with about 6K refusals ("申し訳ありませんが、。。。") filtered out from the category dataset

Lightblue's suite of Kujira datasets (unreleased)
Lightblue's own question-based datasets (unreleased)
Lightblue's own category-based datasets (unreleased)
OASST (Japanese chats only)
ShareGPT (Japanese chats only)
augmxnt/ultra-orca-boros-en-ja-v1 (['airoboros', 'slimorca', 'ultrafeedback', 'airoboros_ja_new'] only)

Developed by

Engineers

Peter Devine

Sho Higuchi

Advisors

Yuuki Yamanaka

Atom Sonoda

Project manager

Shunichi Taniguchi

Dataset evaluator

Renju Aoki

Demo ・ モデルのデモ