Instructions to use QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF", dtype="auto")

llama-cpp-python

How to use QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF",
	filename="Llama-3-8B-Instruct-Finance-RAG.Q2_K.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M

Use Docker

docker model run hf.co/QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M

SGLang

How to use QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF with Ollama:
```
ollama run hf.co/QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M
```

Unsloth Studio new

How to use QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF to start chatting

Docker Model Runner
How to use QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF with Docker Model Runner:
```
docker model run hf.co/QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M
```

Lemonade

How to use QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Llama-3-8B-Instruct-Finance-RAG-GGUF-Q4_K_M

List all available models

lemonade list

QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF

This is quantized version of curiousily/Llama-3-8B-Instruct-Finance-RAG created using llama.cpp

Original Model Card

Llama 3 8B Instruct (Financial RAG)

This model is a fine-tuned version of the original Llama 3 8B Instruct model on 4000 examples from the virattt/financial-qa-10K dataset.

The model is fine-tuned using a LoRA adapter for RAG use cases. It is optimized to answer a question based on a context:

Answer the question:
{question}

Using the information:
{context}

Usage

Load the model:

MODEL_NAME = "curiousily/Llama-3-8B-Instruct-Finance-RAG"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    device_map="auto"
)

pipe = pipeline(
    task="text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=128,
    return_full_text=False,
)

Format the prompt (uses the original Instruct prompt format):

prompt = """
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Use only the information to answer the question<|eot_id|><|start_header_id|>user<|end_header_id|>

How much did the company's net earnings amount to in fiscal 2022?

Information:

```
Net earnings were $17.1 billion in fiscal 2022.
```<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""

And make a prediction:

print(outputs[0]["generated_text"])

$17.1 billion

Here's a helper function to build your prompts:

def create_test_prompt(data_row):
    prompt = dedent(f"""
    {data_row["question"]}

    Information:

    ```
    {data_row["context"]}
    ```
    """)
    messages = [
        {"role": "system", "content": "Use only the information to answer the question"},
        {"role": "user", "content": prompt},
    ]
    return tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

Where data_row must be a dict:

data_row = {
  "question": "...",
  "context": "..."
}

Sample Predictions

Here's a sample of the predictions from trained and untrained models

Example 1

answer: Delta Air Lines' agreements with its regional carriers typically last at least ten years with options for extensions. Delta controls operational aspects like scheduling and pricing.

trained: Delta Connection agreements typically last at least ten years with options for extensions.

untrained: According to the information, the terms of Delta Air Lines' agreements with its regional carriers through Delta Connection are:

1. The agreements typically last at least ten years.
2. There are options for extensions.
3. Delta controls major operational aspects like scheduling and pricing.
4. The regional carriers supply the services.

Example 2

answer: The company evaluates acquisition-related intangibles for impairment by comparing the asset's carrying amount to undiscounted future net cash flows expected from the asset. An impairment loss is recognized if the carrying amount exceeds the asset's recoverable amount.

trained: The company evaluates acquisition-related intangible and other long-lived assets for impairment whenever events or changes in circumstances indicate that the carrying amount may not be recoverable. This involves measuring the recoverability of the asset by comparing its carrying amount to the future undiscounted net cash flows expected to be generated by the asset group.

untrained: According to the information, the company uses the following criteria to determine whether an impairment loss should be recognized on acquisition-related intangible assets:

* Events or changes in circumstances that indicate the carrying amount of an asset may not be recoverable.
* A comparison of the carrying amount of an asset to future undiscounted net cash flows expected to be generated by the asset group.

In other words, the company evaluates whether the asset's carrying amount is recoverable by comparing it to the expected future cash flows, which involves assumptions about future prospects and computations of estimated future cash flows.

Example 3

answer: In the United States, the approval process for biosimilars is governed by the Public Health Service Act (PHSA) and the regulations implementing these statutes, specifically including provisions made under federal health care reform legislation enacted in March 2010.

trained: The Federal Food, Drug, and Cosmetic Act (the FFDCA) and the Public Health Service Act (PHSA)

untrained: The legal framework that governs the approval process for biosimilars in the United States is the Federal Food, Drug, and Cosmetic Act (FFDCA) and the Public Health Service Act (PHSA), as well as the regulations implementing these statutes.

Example 4

answer: Timothy S. Teter holds a B.S. degree in Mechanical Engineering from the University of California at Davis and a J.D. degree from Stanford Law School.

trained: B.S. in Mechanical Engineering from the University of California at Davis and a J.D. from Stanford Law School

untrained: According to the information, Timothy S. Teter holds:

1. A B.S. degree in Mechanical Engineering from the University of California at Davis.
2. A J.D. degree from Stanford Law School.

Example 5

answer: Beginning in fiscal year 2024, the company plans to exclude paused Connected Fitness subscriptions from its new 'Ending Paid Connected Fitness Subscriptions' metric and will treat a pause action as a churn event in its 'Average Net Monthly Paid Connected Fitness Subscription Churn' metric.

trained: Starting in fiscal year 2024, the company will no longer include paused Connected Fitness subscriptions in their Ending Paid Connected Fitness Subscriptions metric and will treat a pause action as a churn event in their Average Net Monthly Paid Connected Fitness Subscription Churn.

untrained: Starting in fiscal year 2024, the company will:

* No longer include paused Connected Fitness subscriptions in the Ending Paid Connected Fitness Subscriptions metric
* Treat a pause action as a churn event in the Average Net Monthly Paid Connected Fitness Subscription Churn

License

Uses the original Llama 3 License. A custom commercial license is available at: https://llama.meta.com/llama3/license

Downloads last month: 427

GGUF

Model size

8B params

Architecture

llama

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Model tree for QuantFactory/Llama-3-8B-Instruct-Finance-RAG-GGUF

Base model

meta-llama/Meta-Llama-3-8B-Instruct

Quantized

(272)

this model

QuantFactory
/

Llama-3-8B-Instruct-Finance-RAG-GGUF