Borea
Collection
3 items โข Updated โข 3
[BOREA model card]
[Model Information]
Based on phi-3.5-mini-Instruct, this model is a general-purpose model with improved performance from the base model after employing multiple tuning methods. In particular, Japanese language performance has been improved.
phi-3.5-mini-Instructใใใผในใจใใฆใ่คๆฐใฎใใฅใผใใณใฐๆๆณใๆก็จใฎใใใๆฑ็จ็ใซใใผในใขใใซใใๆง่ฝใๅไธใใใใขใใซใงใใ็นใซๆฅๆฌ่ชๆง่ฝใๅไธใใฆใใพใใ
[Benchmark Results]
TODO:
ๆจๅฅจใใใไฝฟ็จใฌใคใใฉใคใณ / Recommended Usage Guidelines
ๅ็จๅฉ็จ: ๆฌใขใใซใๅ็จ็ฎ็ใงไฝฟ็จใใๅ ดๅใinfo@axcxept.com ใธใฎใกใผใซ้ฃ็ตกใๅผทใๆจๅฅจใใพใใใใใซใใใใขใใซใฎๅฟ็จใๆนๅใซใคใใฆใฎๅๅใฎๆฉไผใ็ใพใใๅฏ่ฝๆงใใใใพใใ
ใฏใฌใธใใ่กจ่จ: ๆฌใขใใซใไฝฟ็จใพใใฏๆนๅคใใ้ใฏใไปฅไธใฎใใใชใฏใฌใธใใ่กจ่จใ่กใใใจใๆจๅฅจใใพใ๏ผ "This project utilizes HODACHI/Borea-Phi-3.5-mini-Instruct-Jp, a model based on Phi-3.5-mini-Instruct and fine-tuned by Axcxept co., ltd."
ใใฃใผใใใใฏ: ใขใใซใฎไฝฟ็จ็ต้จใซ้ขใใใใฃใผใใใใฏใๆญ่ฟใใพใใinfo@axcxept.com ใพใงใ้ฃ็ตกใใ ใใใ
ใใใใฏๆจๅฅจไบ้ ใงใใใๆณ็่ฆไปถใงใฏใใใพใใใ
Commercial Use: If you plan to use this model for commercial purposes, we strongly encourage you to inform us via email at info@axcxept.com. This allows for potential collaboration on model applications and improvements.
Attribution: When using or adapting this model, we recommend providing attribution as follows: "This project utilizes HODACHI/Borea-Phi-3.5-mini-Instruct-Jp, a model based on Phi-3.5-mini-Instruct and fine-tuned by Axcxept co., ltd."
Feedback: We welcome any feedback on your experience with the model. Please feel free to email us at info@axcxept.com.
Please note that these are recommendations and not legal requirements.
[Usage]
Here are some code snippets to quickly get started with the model. First, run:
pip install flash_attn==2.5.8
pip install accelerate==0.31.0
pip install transformers==4.43.0
pip install -U trl
pip install pytest
Then, copy the snippet from the relevant section for your use case.
ไปฅไธใซใใขใใซใฎๅฎ่กใ็ด ๆฉใ้ๅงใใใใใฎใณใผใในใใใใใใใใคใ็ดนไปใใพใใ ใพใใ
pip install flash_attn==2.5.8
pip install accelerate==0.31.0
pip install transformers==4.43.0
pip install -U trl
pip install pytest
ใๅฎ่กใใไฝฟ็จไพใซ้ข้ฃใใใปใฏใทใงใณใฎในใใใใใใณใใผใใฆใใ ใใใ
[Chat Template]
<|system|>
ใใชใใฏๆฅๆฌ่ช่ฝๅใ้ซใ้ซๅบฆใชAIใงใใ็นๅฅใชๆ็คบใใชใ้ใๆฅๆฌ่ชใง่ฟ็ญใใฆใใ ใใใ<|end|>
<|user|>
ใ็ใ็ฉใใถใคใใผใใจใใ่ทๆฅญใใใใพใใใใใฏใ่ชๅใ่ใใใชใชใธใใซใฎ็ใ็ฉใใใถใคใณใใๅฎ้ใซDNAใ็ทจ้ใใฆไฝใๅบใไปไบใงใใใใชใใ็ใ็ฉใใถใคใใผใงใใๅ ดๅใใฉใใช็ใ็ฉใไฝใใใใงใใ๏ผใพใใใใฎ็ใ็ฉใๆใค็นๅพดใ่ฝๅใซใคใใฆ่ชฌๆใใฆใใ ใใใ
<|end|>
<|assistant|>
Loading the model locally
After obtaining the Phi-3.5-mini-instruct model checkpoint, users can use this sample code for inference.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
torch.random.manual_seed(0)
model = AutoModelForCausalLM.from_pretrained(
"HODACHI/Borea-Phi-3.5-mini-Instruct-Jp",
device_map="cuda",
torch_dtype="auto",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("HODACHI/Borea-Phi-3.5-mini-Instruct-Jp")
messages = [
{"role": "system", "content": "ใใชใใฏๆฅๆฌ่ช่ฝๅใ้ซใ้ซๅบฆใชAIใงใใ็นๅฅใชๆ็คบใใชใ้ใๆฅๆฌ่ชใง่ฟ็ญใใฆใใ ใใใ"},
{"role": "user", "content": "ใ็ใ็ฉใใถใคใใผใใจใใ่ทๆฅญใใใใพใใใใใฏใ่ชๅใ่ใใใชใชใธใใซใฎ็ใ็ฉใใใถใคใณใใๅฎ้ใซDNAใ็ทจ้ใใฆไฝใๅบใไปไบใงใใใใชใใ็ใ็ฉใใถใคใใผใงใใๅ ดๅใใฉใใช็ใ็ฉใไฝใใใใงใใ๏ผใพใใใใฎ็ใ็ฉใๆใค็นๅพดใ่ฝๅใซใคใใฆ่ชฌๆใใฆใใ ใใใ"},
]
pipe = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
)
generation_args = {
"max_new_tokens": 1024,
"return_full_text": False,
"temperature": 0.0,
"do_sample": False,
}
output = pipe(messages, **generation_args)
print(output[0]['generated_text'])
Notes: If you want to use flash attention, call AutoModelForCausalLM.from_pretrained() with attn_implementation="flash_attention_2"
[Model Data]
Training Dataset]
We extracted high-quality data from Japanese Wikipedia and FineWeb to create instruction data. Our innovative training approach allows for performance improvements across various languages and domains, making the model suitable for global use despite its focus on Japanese data.
ๆฅๆฌ่ชใฎWikiใใผใฟใใใณใFineWebใใ่ฏ่ณชใชใใผใฟใฎใฟใๆฝๅบใใInstructionใใผใฟใไฝๆใใพใใใใใฎใขใใซใงใฏๆฅๆฌ่ชใซ็นๅใใใฆใใพใใใไธ็ไธญใฎใฉใใชใฆใผในใฑใผในใงใๅฉ็จๅฏ่ฝใชใขใใญใผใใงใใ
https://huggingface.co/datasets/legacy-datasets/wikipedia https://huggingface.co/datasets/HuggingFaceFW/fineweb
Data Preprocessing
We used a plain instruction tuning method to train the model on exemplary responses. This approach enhances the model's ability to understand and generate high-quality responses across various languages and contexts.
ใใฌใคใณในใใฉใฏใใใฅใผใใณใฐๆๆณใ็จใใฆใๆจก็ฏ็ๅ็ญใๅญฆ็ฟใใใพใใใใใฎๆๆณใซใใใใขใใซใฏๆงใ ใช่จ่ชใใณใณใใญในใใซใใใฆ้ซๅ่ณชใชๅฟ็ญใ็่งฃใ็ๆใใ่ฝๅใๅไธใใฆใใพใใ
Implementation Information
[Pre-Instruction Training]
https://huggingface.co/instruction-pretrain/instruction-synthesizer
[Disclaimer]
ใใฎใขใใซใฏ็ ็ฉถ้็บใฎใฟใ็ฎ็ใจใใฆๆไพใใใใใฎใงใใใๅฎ้จ็ใชใใญใใฟใคใใจใฟใชใใใในใใขใใซใงใใ ๅๆฅญ็ใชไฝฟ็จใใใใทใงใณใฏใชใใฃใซใซใช็ฐๅขใธใฎ้ ๅใๆๅณใใใใฎใงใฏใใใพใใใ ๆฌใขใใซใฎไฝฟ็จใฏใไฝฟ็จ่ ใฎ่ฒฌไปปใซใใใฆ่กใใใใใฎใจใใใใฎๆง่ฝใใใณ็ตๆใฏไฟ่จผใใใพใใใ Axcxeptๆ ชๅผไผ็คพใฏใ็ดๆฅ็ใ้ๆฅ็ใ็นๅฅใๅถ็บ็ใ็ตๆ็ใชๆๅฎณใใพใใฏๆฌใขใใซใฎไฝฟ็จใใ็ใใใใใชใๆๅคฑใซๅฏพใใฆใใๅพใใใ็ตๆใซใใใใใใไธๅใฎ่ฒฌไปปใ่ฒ ใใพใใใ ๅฉ็จ่ ใฏใๆฌใขใใซใฎไฝฟ็จใซไผดใใชในใฏใๅๅใซ็่งฃใใ่ชๅทฑใฎๅคๆญใงไฝฟ็จใใใใฎใจใใพใใ
[Hardware]
H100PCIe ร 8(Running in 2h)
[We are.]
- Downloads last month
- 141