Instructions to use MarsupialAI/Dumbstral-169B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MarsupialAI/Dumbstral-169B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MarsupialAI/Dumbstral-169B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MarsupialAI/Dumbstral-169B")
model = AutoModelForCausalLM.from_pretrained("MarsupialAI/Dumbstral-169B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use MarsupialAI/Dumbstral-169B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MarsupialAI/Dumbstral-169B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarsupialAI/Dumbstral-169B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/MarsupialAI/Dumbstral-169B

SGLang

How to use MarsupialAI/Dumbstral-169B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MarsupialAI/Dumbstral-169B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarsupialAI/Dumbstral-169B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MarsupialAI/Dumbstral-169B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarsupialAI/Dumbstral-169B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use MarsupialAI/Dumbstral-169B with Docker Model Runner:
```
docker model run hf.co/MarsupialAI/Dumbstral-169B
```

Wow!!!

by Alexandria4114 - opened Nov 26, 2024

Discussion

Alexandria4114

Nov 26, 2024

Hi! The model turned out great! It was my two best and most fun weekends. the model has a good consistency in the logic of actions and is really creative! I loved putting my character in awkward situations and dialogues so much) His answers, and especially mental dialogues with himself, are certainly something! He self-destructed himself. The model describes the situation, objects and creatures in detail very beautifully. But the only thing that the model is very adaptable to previous messages: that is, it turns from a regular chat into something literary... Heh, apparently I just don't like this) This model imitates the character of the character very well. and I would be very glad to see this model in aihorde again at least once!
Sincerely, from Alexandria.

lightning-missile

Dec 22, 2024

Hi! The model turned out great! It was my two best and most fun weekends. the model has a good consistency in the logic of actions and is really creative! I loved putting my character in awkward situations and dialogues so much) His answers, and especially mental dialogues with himself, are certainly something! He self-destructed himself. The model describes the situation, objects and creatures in detail very beautifully. But the only thing that the model is very adaptable to previous messages: that is, it turns from a regular chat into something literary... Heh, apparently I just don't like this) This model imitates the character of the character very well. and I would be very glad to see this model in aihorde again at least once!
Sincerely, from Alexandria.

Can you share your sampler settings? Thanks.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment