KokLLaMA-3.2-3B-Instruct

KokLLaMA-3.2-3B-Instruct is a Low-Rank Adapter (LoRA) fine-tune of the Llama-3.2-3B-Instruct model, designed to understand and generate text in Kokborok (the native language of the Tripuri people).

This model serves as a general-purpose instruction-following assistant for the Kokborok language, capable of answering queries in Math, Science, and Social scenarios, translating concepts, and maintaining conversation in the local dialect used in Tripura, India.

Model Details

• Developer: Agniva Maiti
• Base Model: meta-llama/Llama-3.2-3B-Instruct
• Language: Kokborok (trp)
• Fine-tuning Method: LoRA (Low-Rank Adaptation) via QLoRA
• Precision: 4-bit Quantized Training (NF4)

Training Data

The model was trained on a custom Kokborok Instruction Dataset containing approximately 11,000 instruction-following pairs. The dataset covers diverse categories including:

• General Conversation: Greetings, daily life advice.
• STEM: Mathematics, Physics (Kinematics, Electricity), and Logic.
• Cultural Knowledge: History of Tripura, festivals, and social norms.

Data Splitting: To ensure robust evaluation, the dataset was split as follows during training:

• Training: 95%
• Test: 5% (held-out for evaluation loss monitoring)

Training Hyperparameters

The model was fine-tuned using the SFTTrainer with the following configuration:

• Epochs: 3
• Batch Size: 8 (per device) with 2 gradient accumulation steps
• Learning Rate: 2e-4
• Optimizer: paged_adamw_32bit
• Sequence Length: 1024
• LoRA Config:
  • Rank (r): 64
  • Alpha: 128
  • Dropout: 0.05
  • Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Intended Use

This model is intended for:

• Chatbots and assistants requiring Kokborok language support.
• Educational tools for students studying in Kokborok medium.
• Translation assistance between English and Kokborok.

How to Use

You can use this model with the peft and transformers libraries.

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# 1. Load Base Model
base_model_id = "meta-llama/Llama-3.2-3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# 2. Load KokLLaMA Adapter
adapter_id = "agnivamaiti/KokLLaMA-3.2-3B-Instruct"
model = PeftModel.from_pretrained(model, adapter_id)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

# 3. Define the Prompt (Llama 3 Format)
question = "Kokborok o 'Good Morning' tamo hwnwi sao?"
prompt = (
    f"<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n"
    f"{question}<|eot_id|>"
    f"<|start_header_id|>assistant<|end_header_id|>\n\n"
)

# 4. Inference
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
    **inputs,
    max_new_tokens=150,      
    do_sample=True,
    temperature=0.3,
    top_k=15,
    top_p=0.3,
    repetition_penalty=1.2,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True).split("assistant")[-1].strip())

Limitations & Safety

• Hallucinations: Like all LLMs, this model may generate incorrect information.
• Bias: The model inherits biases from the base Llama 3.2 model and the specific dialectal patterns found in the training data.
• Critical Use: Not suitable for medical, legal, or financial advice.

Credits

• Acknowledgments: Special thanks to the friends from Tripura who validated the dataset and model outputs.