Instructions to use BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny")
model = AutoModelForCausalLM.from_pretrained("BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny

SGLang

How to use BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny with Docker Model Runner:
```
docker model run hf.co/BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny
```

How much 3090-hours did you need for the training epoch?

by UMCU - opened Nov 21, 2023

Discussion

UMCU

Nov 21, 2023

Hi Bram,

thanks for this great work (and @yhavinga of course)! I have a question. I am in the process of arranging compute to
finetune a Dutch language model of about the same size with about the same sized dataset. You used 4x3090's, do you still remember the wall clock time and the average GPU-loading?

As for the finetuning, I assume you trained all layers?

Groeten,

Bram

BramVanroy

Owner Dec 4, 2023

HI @UMCU . This flew below the radar, sorry for not catching it!

To be honest I do not remember all the details, but if you look at the train_results.json file you can see some metrics there, like the runtime in seconds and how many samples were processed per second at a context length of 4096 (https://huggingface.co/BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny/blob/main/train_results.json#L6).

For finetuning I did not finetune all layers due to limited compute. I used QLoRA. If you have the compute, I would recommend doing a full finetune indeed, or at least LoRA with all linear layers.

Hope that helps!

Bram

BramVanroy changed discussion status to closed Dec 13, 2023

UMCU

Dec 13, 2023

•

edited Dec 13, 2023

Thanks for the reply Bram!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment