Instructions to use latincy/latin-bert with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use latincy/latin-bert with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="latincy/latin-bert")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("latincy/latin-bert") model = AutoModel.from_pretrained("latincy/latin-bert", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Commit History
9 days ago
-
release: v1.2.1 — tokenizers 0.23 BertProcessing fix
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>c559e75
30 days ago
-
release: v1.2.0 — promote no-code tokenizer (drop rc1 prerelease note)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>307ac74
-
release: mark branch as v1.2.0-rc1 prerelease with revision-pinned install note
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>df2aa0e -
feat: pure-JSON tokenizer — trust_remote_code no longer required (v1.2.0)
Ship tokenizer.json built entirely from serializable tokenizers components. Zero-width lookaround Replace normalizers reproduce the tensor2tensor char-class split, escaping, and token-end underscores; WordPiece with an empty continuation prefix reproduces the greedy longest-match. AutoTokenizer.from_pretrained now loads with no flags. Verified ID-identical to the Python reference tokenizer on 5,000+ corpus lines, all 2,187 alphabet characters, and adversarial inputs. Documented divergences (unreachable for real text): out-of-alphabet chars -> [UNK] instead of ordinal escapes; literal U+0001 stripped. Custom Python classes retained as reference implementation and test oracle; trust_remote_code=True loads keep working unchanged. Also: regenerable via scripts/build_tokenizer_json.py (--check enforced in tests), fast-tokenizer word_ids/is_split_into_words tests, model card reworked to cover all four case studies, use_fast=False DEV_PLAN item closed as obsoleted upstream (transformers v5…36c8f15
-
release: v1.1.2 — full-string tokenization fix (eval-confirmed)
Cluster eval (torch, L40S, job 13332650) confirms the fix end-to-end: - infilling n=1161, NATIVE full-string path: P@1=0.331 P@10=0.622 P@50=0.740 — reproduces Bamman & Burns Table 3 exactly, identical to the per-word path. - reported symptoms recovered: Caesar aliam (#2), Livy primo (#1). Bumps version to 1.1.2 and adds the changelog entry (held until the eval was green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>9cfba71
-
fix: add do_lower_case=True to tokenizer (v1.1.1)
The original Latin BERT vocab was trained on lowercased text and all original case studies called .lower() before tokenizing. The HF wrapper was missing this step, causing uppercase characters to be escaped to ASCII codepoints (e.g. C → \67;), inflating token counts ~4x and producing embeddings the model never saw during pretraining. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c7b1be1
-
test: add contextual nearest neighbors case study (Bamman & Burns §4.4)
Three tests: - test_embedding_parity: fast CPU test verifying word-level embeddings - test_generate_embeddings: generates embeddings for Latin Library corpus - test_contextual_nn_queries: runs paper's example queries with soft assertions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>3510517 -
chore: register pytest slow marker
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>bb53c05
-
Initial: HF-compatible Latin BERT tokenizer (Bamman & Burns 2020)
PreTrainedTokenizer wrapping the original tensor2tensor SubwordTextEncoder. Produces identical token IDs to the original (verified against POS case study: Perseus 95.2%, ITTB 99.2%, PROIEL 98.2%). No TensorFlow dependency. 13/13 tests passing: encoding, roundtrip decode, save/load. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>68d8806