Instructions to use Lin-Chen/open-llava-next-llama3-8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Lin-Chen/open-llava-next-llama3-8b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Lin-Chen/open-llava-next-llama3-8b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Lin-Chen/open-llava-next-llama3-8b", dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Lin-Chen/open-llava-next-llama3-8b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Lin-Chen/open-llava-next-llama3-8b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Lin-Chen/open-llava-next-llama3-8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Lin-Chen/open-llava-next-llama3-8b

SGLang

How to use Lin-Chen/open-llava-next-llama3-8b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Lin-Chen/open-llava-next-llama3-8b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Lin-Chen/open-llava-next-llama3-8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Lin-Chen/open-llava-next-llama3-8b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Lin-Chen/open-llava-next-llama3-8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Lin-Chen/open-llava-next-llama3-8b with Docker Model Runner:
```
docker model run hf.co/Lin-Chen/open-llava-next-llama3-8b
```

open-llava-next-llama3-8b Model Card

Model details

Model type: open-llava-next-llama3-8b is an open-source implementation chatbot trained by fine-tuning the entire model on open-source Open-LLaVA-Next-mix1M data.

Model date: open-llava-next-llama3-8b was trained in May 2024.

Paper or resources for more information: [Code]

Name	ViT	LLM	Weights	MME	SEED	SQA	MMB	MMB-CN	TextVQA	GQA
llava-next-vicuna-7b	CLIP-L-336	Vicuna-7B	HF	1519	70.2	70.1	67.4	60.6	64.9	64.2
open-llava-next-vicuna-7b	CLIP-L-336	Vicuna-7B	HF	1540	71.1	70.7	68.5	60.7	67.2	64.3
open-llava-next-llama3-8b	CLIP-L-336	LLaMA3-8B	HF	1552	74.4	77.3	74.4	70.4	69.8	65.9

Usage

You can utilize this model as we provide in our [repository]. Moreover, you can direct load this model and use it in the [LLaVA repository].

Training dataset

All training data are open-sourced in our repository.

ShareGPT4V Mix665K
200K GPT4V-generated instruction data (ALLaVA)
200K various VQA data

Intended use

Primary intended uses: The primary use of open-llava-next-llama3-8b is research on large multimodal models and chatbots.

Primary intended users: The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.

Downloads last month: 16

Safetensors

Model size

8B params

Tensor type

BF16

Model tree for Lin-Chen/open-llava-next-llama3-8b

Finetunes

4 models

Dataset used to train Lin-Chen/open-llava-next-llama3-8b

Space using Lin-Chen/open-llava-next-llama3-8b 1

Collection including Lin-Chen/open-llava-next-llama3-8b

Open-LLaVA-NeXT

Collection

Open-source implementation of the LLaVA-NeXT series with Open-LLaVA-NeXT repositary • 3 items • Updated May 29, 2024 • 4