Instructions to use Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B")
model = AutoModelForCausalLM.from_pretrained("Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B

SGLang

How to use Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B with Docker Model Runner:
```
docker model run hf.co/Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Open-Insurance-LLM-Llama3-8B

This model is a domain-specific language model based on Nvidia Llama 3 ChatQA, fine-tuned for insurance-related queries and conversations. It leverages the architecture of Llama 3 and is specifically trained to handle insurance domain tasks.

Model Details

Model Type: Instruction-tuned Language Model
Base Model: nvidia/Llama3-ChatQA-1.5-8B
Finetuned Model: Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B
Quantized Model: Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B-GGUF
Model Architecture: Llama
Parameters: 8.05 billion
Developer: Raj Maharajwala
License: llama3
Language: English

Quantized Model

Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B-GGUF: https://huggingface.co/Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B-GGUF

Training Data

The model has been fine-tuned on the InsuranceQA dataset using LoRA (8 bit), which contains insurance-specific question-answer pairs and domain knowledge. trainable params: 20.97M || all params: 8.05B || trainable %: 0.26%

LoraConfig(
  r=8,
  lora_alpha=32,
  lora_dropout=0.05,
  bias="none",
  task_type="CAUSAL_LM",
  target_modules=['up_proj', 'down_proj', 'gate_proj', 'k_proj', 'q_proj', 'v_proj', 'o_proj']
)

Model Architecture

The model uses the Llama 3 architecture with the following key components:

8B parameter configuration
Enhanced attention mechanisms from Llama 3
ChatQA 1.5 instruction-tuning framework
Insurance domain-specific adaptations

Files in Repository

Model Files:
- model-00001-of-00004.safetensors (4.98 GB)
- model-00002-of-00004.safetensors (5 GB)
- model-00003-of-00004.safetensors (4.92 GB)
- model-00004-of-00004.safetensors (1.17 GB)
- model.safetensors.index.json (24 kB)
Tokenizer Files:
- tokenizer.json (17.2 MB)
- tokenizer_config.json (51.3 kB)
- special_tokens_map.json (335 Bytes)
Configuration Files:
- config.json (738 Bytes)
- generation_config.json (143 Bytes)

Use Cases

This model is specifically designed for:

Insurance policy understanding and explanation
Claims processing assistance
Coverage analysis
Insurance terminology clarification
Policy comparison and recommendations
Risk assessment queries
Insurance compliance questions

Limitations

The model's knowledge is limited to its training data cutoff
Should not be used as a replacement for professional insurance advice
May occasionally generate plausible-sounding but incorrect information

Bias and Ethics

This model should be used with awareness that:

It may reflect biases present in insurance industry training data
Output should be verified by insurance professionals for critical decisions
It should not be used as the sole basis for insurance decisions
The model's responses should be treated as informational, not as legal or professional advice

Citation and Attribution

If you use this model in your research or applications, please cite:

@misc{maharajwala2024openinsurance,
  author = {Raj Maharajwala},
  title = {Open-Insurance-LLM-Llama3-8B},
  year = {2024},
  publisher = {HuggingFace},
  url = {https://huggingface.co/Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B}
}

Downloads last month: 22

Safetensors

Model size

8B params

Tensor type

F16

Model tree for Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B

Base model

nvidia/Llama3-ChatQA-1.5-8B

Finetuned

(6)

this model

Quantizations

1 model