Rziane/mbert-kreyol-RH

Continued MLM pre-training of bert-base-multilingual-cased on a custom text corpus.

Training

• base model: bert-base-multilingual-cased
• epochs: 8.0
• batch size (per device): 64
• gradient accumulation: 1
• learning rate: 5e-05
• scheduler: cosine (warmup 0.06)
• weight decay: 0.01
• max length: 128
• mlm probability: 0.15
• precision: fp16
• seed: 42

Final eval

• eval_loss: 1.5256
• eval_runtime: 406.4867
• eval_samples_per_second: 584.1000
• eval_steps_per_second: 4.5630
• epoch: 8.0000
• perplexity: 4.5978

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained('Rziane/mbert-kreyol-RH')
model = AutoModelForMaskedLM.from_pretrained('Rziane/mbert-kreyol-RH')