Commit History
5 days ago
-
docs: add MIT LICENSE (copyright line as upstream dbamman/latin-bert)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>05b7e7f -
docs: card license apache-2.0 -> MIT (matches upstream dbamman/latin-bert)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>2a08f55 -
release: v1.2.0 — promote no-code tokenizer (drop rc1 prerelease note)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>307ac74
-
release: mark branch as v1.2.0-rc1 prerelease with revision-pinned install note
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>df2aa0e -
docs: frame OOA->[UNK] divergence as improvement, still reported as difference
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>85f785b -
test+docs: prove case-study replication via ID-identity on the actual datasets
Sweep the shipped case-study inputs through both tokenizers: all 15,444 infilling texts and 47,743 WSD texts bit-identical (zero out-of-alphabet chars in either file), both §4.4 CNN queries identical, and 29,983 Perseus/PROIEL/ITTB sentences identical except one wordform containing U+1F13 (polytonic Greek outside the vocab alphabet — the documented OOA divergence; both tokenizers emit garbage for it). Full 97.5M-char CNN corpus scan: 65 out-of-alphabet char occurrences total, all OCR noise. Added the infilling/WSD sweep as a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>dec104b -
feat: pure-JSON tokenizer — trust_remote_code no longer required (v1.2.0)
Ship tokenizer.json built entirely from serializable tokenizers components. Zero-width lookaround Replace normalizers reproduce the tensor2tensor char-class split, escaping, and token-end underscores; WordPiece with an empty continuation prefix reproduces the greedy longest-match. AutoTokenizer.from_pretrained now loads with no flags. Verified ID-identical to the Python reference tokenizer on 5,000+ corpus lines, all 2,187 alphabet characters, and adversarial inputs. Documented divergences (unreachable for real text): out-of-alphabet chars -> [UNK] instead of ordinal escapes; literal U+0001 stripped. Custom Python classes retained as reference implementation and test oracle; trust_remote_code=True loads keep working unchanged. Also: regenerable via scripts/build_tokenizer_json.py (--check enforced in tests), fast-tokenizer word_ids/is_split_into_words tests, model card reworked to cover all four case studies, use_fast=False DEV_PLAN item closed as obsoleted upstream (transformers v5…36c8f15 -
docs: note full case-study reproduction (incl. contextual-NN) in v1.1.2 changelog
The §4.4 contextual nearest neighbors case study now reproduces through the native path (Tables 6–7: in 0.835, audentes 0.919, lexical variants audentis/audaces/audentem retrieved), alongside POS/WSD. No behavior or version change — documentation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>e71173a -
test: make contextual-NN case study faithful to paper §4.4
The audentes query asserted `exact-string 'audentes' >= 10`, which contradicts Bamman & Burns (2020) Table 7 — whose neighbors deliberately span lexical variants (audentis, audaces, audentem) and intertextual reuse. Use the paper's exact two queries (in, audentes; drop the interpolated amor) and assert what Tables 6 & 7 actually show: a strong top neighbor plus, for audentes, cross-variant retrieval. GPU-verified: in top cos 0.835 (paper 0.835), audentes 0.919 (paper 0.926), variants audaces/audentem/audentis retrieved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>a313dd2
-
release: v1.1.2 — full-string tokenization fix (eval-confirmed)
Cluster eval (torch, L40S, job 13332650) confirms the fix end-to-end: - infilling n=1161, NATIVE full-string path: P@1=0.331 P@10=0.622 P@50=0.740 — reproduces Bamman & Burns Table 3 exactly, identical to the per-word path. - reported symptoms recovered: Caesar aliam (#2), Livy primo (#1). Bumps version to 1.1.2 and adds the changelog entry (held until the eval was green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>9cfba71 -
test: full-string eval harness + reporter reproduction
Adds end-to-end coverage for the native full-string tokenization path that the space-escape bug corrupted (previously only word-by-word paths were exercised): - test_infilling: parametrize over tokenize mode {per_word, native}; both must reproduce published P@1/@10/@50. "native" = the documented tokenizer(text) usage. - test_bug_reports: direct reproduction of the two reported failures via the native masked-LM path. Verified locally on the fixed model: Caesar "...Belgae, [MASK] Aquitani" -> aliam ranks #2 (top-10: alteram, aliam, secundam, ...); Livy "...qui [MASK] Punico bello" -> primo ranks #1. Both symptoms resolved. Cluster driver (cluster/eval_fullstring.sbatch + cluster/smoke_eval.py) is kept untracked (cluster/ is gitignored: machine-specific paths). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>f93e321 -
fix: whitespace-split before subword encoding (full-string tokenization)
The HF tokenizer's full-string path char-class-split the entire string without dropping inter-word spaces (unlike tensor2tensor tokenizer.encode and gen_berts.py's LatinTokenizer, which split on whitespace first). Every space was escaped to `\32;_`, injecting 3 spurious subtokens between words — input the model never trained on. This corrupted masked-word predictions and broke subword->word alignment for the documented `tokenizer("sentence")` usage, while the word-by-word case-study path stayed correct (which is why validation passed). Reported in two independent bug reports. Fix (slow + root copy + fast): split on whitespace first, then char-class encode each word. Native full-string output now equals the original per-word tokenization (verified: slow == fast IDs). Decode renders word boundaries as spaces (BERT-style; "tres." -> "tres ."). Also fixes a latent fast-tokenizer bug: it ignored do_lower_case (no Rust normalizer), escaping capitals to codepoints (`Gallia` -> `\71;allia`) …b8d6c6c
-
fix: apply do_lower_case to root-level HF-facing files
The v1.1.1 commit only updated the src/ package copies. The root-level tokenization_latin_bert.py and tokenizer_config.json (which HF and the cluster actually serve) were not updated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>f25becd -
fix: add do_lower_case=True to tokenizer (v1.1.1)
The original Latin BERT vocab was trained on lowercased text and all original case studies called .lower() before tokenizing. The HF wrapper was missing this step, causing uppercase characters to be escaped to ASCII codepoints (e.g. C → \67;), inflating token counts ~4x and producing embeddings the model never saw during pretraining. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c7b1be1
-
fix: revert tie_word_embeddings — safetensors needs tying for decoder weights
The safetensors conversion de-duplicates tied weights, storing only one copy. With tie_word_embeddings=false, the decoder weights were missing and randomly initialized, breaking fill-mask predictions. The default tying behavior correctly reconstructs them from the shared embedding, and the original warning disappears since safetensors has no duplicates. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>8c959a5 -
Adding `safetensors` variant of this model (#1)
- Adding `safetensors` variant of this model (a6d5a2ae7e272f798ea67794367e728d7419c405) Co-authored-by: Safetensors convertbot <SFconvertbot@users.noreply.huggingface.co>a533a28 -
chore: gitignore cluster/, track test vocab fixture
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>a285690 -
fix: add decode/unescape to fast tokenizer, silence tied-weights warning
The fast tokenizer (loaded by default via AutoTokenizer) was missing decode logic, so t2t escape sequences and trailing underscore word boundary markers were passed through raw (e.g. "est_" instead of "est"). Added _unescape(), convert_tokens_to_string(), and _decode() methods. Also added tie_word_embeddings=false to config.json to silence the BertForMaskedLM tied-weights warning. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>86e0990 -
docs: remove blockquote from experimental note
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>7a1b678 -
docs: add links to original repo and paper, add experimental proviso
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>0f1214f
-
feat: add LatinBertTokenizerFast with word_ids() support
Builds a Rust-backed tokenizers.Tokenizer with custom pre-tokenizer for char-class split + escaping, WordPiece model (greedy longest-match), and BertProcessing post-processor. Provides native word_ids() needed by Flair and Stanza for subword-to-word alignment. Verified: 18/18 equivalence tests pass (slow vs fast), all existing tokenizer tests pass, is_split_into_words mode works correctly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>ed6af90
-
chore: re-upload model weights (pytorch_model.bin)
6d20995 -
chore: add HF model repo files (config, tokenizer, encoder, README)
Adds the files needed for HuggingFace AutoModel/AutoTokenizer loading: - config.json (BERT config) - tokenization_latin_bert.py (custom tokenizer at root for trust_remote_code) - tokenizer_config.json (auto_map for tokenizer class) - latin.subword.encoder (vocabulary file) - .gitattributes (HF LFS config) - README.md (HF model card) Model weights (pytorch_model.bin) to be uploaded separately. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>872519e -
refactor: extract shared case study utils and move data to tracked paths
- Extract shared BertForSequenceLabeling, get_batches, word_to_subtokens into tests/case_study_utils.py to reduce duplication across test files - Move WSD and infilling data from .claude/reference/ (gitignored) to data/case_studies/ so they ship with the HF repo - Update conftest.py default model path to latincy/latin-bert - Add scripts/benchmark.py (model-agnostic benchmark runner) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>f04d50f
-
fix: handle >512 token sentences and add MPS device support
- Skip sentences exceeding BERT's 512 max sequence length - Add _get_device() helper: CUDA > MPS > CPU - Fix embedding generation crashing on long Latin Library texts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>2c07f6c -
test: add contextual nearest neighbors case study (Bamman & Burns §4.4)
Three tests: - test_embedding_parity: fast CPU test verifying word-level embeddings - test_generate_embeddings: generates embeddings for Latin Library corpus - test_contextual_nn_queries: runs paper's example queries with soft assertions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>3510517 -
feat: make benchmarks model-agnostic with --model-path option
Add conftest.py with --model-path pytest option so the same tests can benchmark any HF-compatible model. Add cluster/benchmark.sbatch for GPU runs on the cluster. Usage: pytest tests/ -m slow --model-path /path/to/model sbatch benchmark.sbatch /path/to/model wsd Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>8af2caa -
chore: register pytest slow marker
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>bb53c05 -
test: add WSD case study reproduction (Bamman & Burns Table 2)
Binary sense classification with 10-fold CV across 201 lemmas. Validates accuracy against reference 75.4% with +/-2% tolerance. Uses latin.sense.data from original repo. Trains a separate BertForSequenceLabeling model per lemma for up to 100 epochs, accumulates dev/test correct counts across all lemmas, and picks the best dev epoch to report test accuracy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>73784ba -
test: add POS tagging case study reproduction (Bamman & Burns Table 1)
Fine-tunes linear classifier on BERT for POS tagging on three UD Latin treebanks. Validates accuracy against reference: Perseus 94.3%, PROIEL 98.2%, ITTB 98.8% with +/-1% tolerance. Perseus verified at 95.2%. Handles tokenization by processing each word individually through the subword encoder (matching original LatinTokenizer behavior) and treats [CLS]/[SEP] as special tokens that bypass subword encoding. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>bbde973 -
test: add infilling case study reproduction (Bamman & Burns Table 3)
Reproduces the masked word prediction experiment using BertForMaskedLM. Validates Precision@1 (0.331), @10 (0.622), @50 (0.740) against reference with +/-1% tolerance. Uses emendation_filtered.txt from original repo. Key porting notes vs original predict_word.py: - Tokenize word-by-word with lowercasing (matching cltk WordTokenizer) - Insert predicted subtoken before [MASK] for multi-subtoken words - Find [MASK] position dynamically in _proc() (matching original) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>c5bfe4c -
Fix tokenizer ID offset: reserve IDs 0-4 for BERT special tokens
The original Latin BERT model expects [PAD]=0, [UNK]=1, [CLS]=2, [SEP]=3, [MASK]=4 with SubwordTextEncoder IDs shifted by +5. This commit adds those five special tokens, shifts all subtoken IDs accordingly, and implements build_inputs_with_special_tokens so encode(add_special_tokens=True) wraps with [CLS]/[SEP]. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>ce59834
-
Initial: HF-compatible Latin BERT tokenizer (Bamman & Burns 2020)
PreTrainedTokenizer wrapping the original tensor2tensor SubwordTextEncoder. Produces identical token IDs to the original (verified against POS case study: Perseus 95.2%, ITTB 99.2%, PROIEL 98.2%). No TensorFlow dependency. 13/13 tests passing: encoding, roundtrip decode, save/load. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>68d8806