Changelog: 0.5.x

Archived verbatim from CHANGELOG.md. Newer releases are in the changelog; the archive index lists every older series.

[0.5.0] — 2026-06-06

Added

  • Context-aware transliteration for abjad scripts (Arabic, Persian, Hebrew). transliterate(text, context=True) uses dictionary-based vowel restoration with bigram context disambiguation to produce readable romanized text instead of consonant skeletons.
  • Arabic: Tashkeela corpus (65.7M words), 182K unigrams + 200K bigrams. Covers 99%+ of newspaper vocabulary.
  • Hebrew: Project Ben Yehuda corpus (11.4M words), 227K unigrams + 200K bigrams. Covers literary Hebrew.
  • Persian: 266 curated common words + optional Wiktionary expansion (14.9K entries available via harvester script).
  • list_context_langs(): returns language codes that support context=True (currently ["ar", "fa", "he"]).
  • LangMeta.context field: "full", "partial", or "none" — enables web/WASM clients to show/hide a context toggle per language.
  • ScriptMeta.context_aware field: bool — enables toggle per detected script.
  • Dictionary build tooling:
  • scripts/build_arabic_dict.py — corpus-based Arabic dictionary builder
  • scripts/build_hebrew_dict.py — corpus-based Hebrew dictionary builder
  • scripts/build_persian_dict.py — curated vocabulary Persian builder
  • scripts/harvest_wiktionary_persian.py — Wiktionary Persian harvester
  • scripts/bootstrap_dicts.sh — reproducible bootstrap from zero with pinned checksums. All parameters auditable, no manual steps.
  • Abjad transliteration documentation (docs/user-guide/abjad-transliteration.md) covering all three languages, standards used, comparison with other systems.
  • pip extras: pip install translit-rs[arabic], [hebrew], [context] for optional context dictionary installation.
  • Rust context engine (src/context.rs): binary dictionary reader, Arabic/Hebrew tokenizer, three-tier resolve (bigram → unigram → context-free fallback), lazy-loaded global singletons via OnceLock.
  • 28 context-aware tests (8 Arabic, 14 Persian, 6 Hebrew).

Changed

  • Repositioning (docs + metadata only — no API or coverage changes). The project now leads with its differentiated, proven core: Unicode adversarial-text defense and canonicalization (TR39 visual confusable mapping), with standards-based Latin/Cyrillic/Greek transliteration as the supporting pillar and CJK/Indic/other scripts framed as best-effort, unidecode-compatible coverage.
  • Rewrote the package description, keywords, and classifiers (added Topic :: Security) across pyproject.toml, Cargo.toml, and mkdocs.yml to surface the security use case for discovery.
  • Restructured README.md / docs/index.md to lead with defense; introduced an explicit three-tier coverage model (core / compatibility / best-effort).
  • Added an Adversarial-Text Defense guide (docs/security/adversarial-defense.md) documenting the phonetic-vs-visual distinction, the XMR metric, and benchmark evidence; elevated security to a top-level docs navigation section.
  • Reframed the Unidecode migration guide: the unidecode alias is for romanization compatibility, not security (it cannot reverse homoglyph attacks).

Fixed

  • Linux x86_64 wheels are now built as cp39-abi3 instead of a version-specific cp38-cp38 wheel. Previously the only published x86_64 Linux wheel targeted CPython 3.8, so pip fell back to a source build (requiring a Rust toolchain) on Linux x86_64 for Python 3.9+. The publish workflow now pins the build interpreter and guards against the regression. (#26)
  • Documentation: corrected the built-in language-profile count (inconsistently reported as 64 in one place; now consistently 83), and fixed several homoglyph code examples whose expected output was wrong (e.g. leading-character ordering in strip_obfuscation examples). All README/doc examples are now verified against the built library.

Security

  • Pinned all third-party GitHub Actions to commit SHAs across the CI and release workflows (resolves the CodeQL actions/unpinned-tag findings) and added .github/dependabot.yml to keep them current. This hardens the release pipeline, which uses PyPI trusted publishing (id-token: write).
  • Bumped dev/docs dependencies flagged by Dependabot: Pygments → 2.20.0 and pytest → 9.0.3 (the pytest bump applies on Python ≥ 3.10; Python 3.9 stays on pytest 8.4.2, since pytest 9 requires ≥ 3.10). Both are development-only — the package has no runtime dependencies.

Notes

  • No public API, language registry, or script coverage was removed. All existing imports, language codes, and the pinned API surface are unchanged.