latin-bert / tokenizer_config.json

Claude Opus 4.6 (1M context)

feat: add LatinBertTokenizerFast with word_ids() support

ed6af90 7 months ago

364 Bytes

	{
	"tokenizer_class": "LatinBertTokenizer",
	"auto_map": {
	"AutoTokenizer": ["tokenization_latin_bert.LatinBertTokenizer", "tokenization_latin_bert_fast.LatinBertTokenizerFast"]
	},
	"pad_token": "[PAD]",
	"unk_token": "[UNK]",
	"cls_token": "[CLS]",
	"sep_token": "[SEP]",
	"mask_token": "[MASK]",
	"eos_token": "<EOS>_",
	"model_max_length": 512
	}