Changelog: 0.11.x

Archived verbatim from CHANGELOG.md. Newer releases are in the changelog; the archive index lists every older series.

[0.11.1] — 2026-07-13

Fixed

  • ml_normalize is now idempotent on NFKD-exposed symbol bases (#498). The "cldr" preset was non-idempotent on negated-relation symbols whose NFKD decomposition strips a combining overlay to expose a nameable base (e.g. ≇ U+2247 → ≅ U+2245 + U+0338 overlay). Because demojize ran before accent-stripping, the freshly-exposed base was only named on a second call, so ml_normalize(x) != ml_normalize(ml_normalize(x)) across the enumerated 17-member negated-symbol class. A second CLDR demojize pass now runs right after accent-stripping, so bases exposed within a call reach a true fixed point in a single call.
  • normalize_confusables is now idempotent and complete on confusable + combining-mark input (#523). Confusable folding and canonical composition interact both ways — a fold can expose a composition (¥+◌̀ → Y+◌̀ → Ỳ) and a composition can expose a new fold (Ҫ+◌̧ → Ç → C, since Ç is itself a confusable) — so a single pass was not always a fixed point, and could even leave a residual confusable in the output. The fold/compose pass now iterates to a fixed point, restoring both idempotency and completeness. Guarded by a new exhaustive Tier-3 test over every confusable × combining-mark pair (~9M).

Documentation

  • Documented unidecode() Cyrillic soft/hard-sign collisions (#511). The compatibility unidecode() path maps the Cyrillic soft sign (ь) and hard sign (ъ) to the empty string, so otherwise-distinct inputs can collide; the limitation is now called out with a pointer to the script-aware transliterate(…, lang=…) path that preserves them.

Internal

  • Binding publishers are gated on the published core (#500). The npm and RubyGems publish jobs build their native addon/gem against disarm as a crates.io dependency, so on a release they now wait for the core crate to land on crates.io before building instead of racing it — removing the manual re-run every prior release required. No library-behavior change.
  • Held phf/phf_codegen at 0.13 to preserve MSRV 1.81 (#510). phf 0.14 moves to edition 2024 / Rust 1.85; a dependabot group now keeps the pair in lockstep and pins it below 0.14 until the MSRV is deliberately raised.

[0.11.0] — 2026-06-21

Performance

  • Transliterate recovers most of the form-invariance compose-at-lookup regression. The #475/#477/#481 boundary added a needs_composition pre-scan that walked every non-ASCII input a second time (UTF-8 decode + is_combining_mark trie lookup per character) before transliterating, so the hot path paid two full passes where it used to pay one — the latin/unidecode comparator ratio fell ~18 → ~11 across #474 → #480. The mark/jamo detection is now fused into the engine's existing decode loop: the fast pass bails to the compose path only when it actually meets a combining mark, so mark-free input (the common case) makes a single pass. needs_composition (still used by the confusables fold) also drops the trie lookup for a range fast-path over U+0000–058F. Behaviour is identical (verified by the exhaustive, formal, and form-invariance suites); Rust-level micro-bench gains, pre → post: latin −33%, cyrillic −20%, mixed −21%, greek −12% ns/char.

Added

  • Strip invisible & non-interchange code points in the security presets (#413). The presets a service puts in front of an LLM, a logger, or a denylist now neutralize the dominant 2024–25 "ASCII smuggling" channels and the adjacent non-interchange classes that survive NFKC and the existing zero-width passes: the Unicode Tags block (U+E0000–U+E007F, including the previously-missed U+E0001), variation selectors, the Combining Grapheme Joiner (U+034F, a denylist-evasion blocker), noncharacters, and the Private Use Area; the Braille Pattern Blank (U+2800) now folds to a space rather than surviving as invisible padding. None of this is a blanket delete — a well-formed emoji subdivision flag (U+1F3F4 … U+E007F) is preserved, and display_clean keeps the VS15/VS16 presentation selectors after a base and preserves the PUA (icon fonts), while the comparison presets (security_clean, normalize_user_input, strip_obfuscation) strip it. Four standalone helpers — strip_tags, strip_variation_selectors, strip_noncharacters, strip_pua — are exposed across the Rust core and the Python, Node, and Ruby bindings for composing policy directly. Output change: the comparison presets now remove these classes; idempotency is preserved (a terminal NFC recomposes any base+mark adjacency a strip creates).

  • Bidi-direction conflict detection (has_bidi_conflict, #412). A new primitive that flags text mixing strong left-to-right and strong right-to-left characters — the precondition for Unicode Bidi display-reordering and the structural signal behind "BiDi Swap"-style spoofs (an LTR brand label stacked on an RTL domain, varonis.com.ו.קום). Unlike a U+202x override check, it fires on the real letters. Derived from disarm's own script ranges (no new table); exposed across the Rust core (disarm::api::has_bidi_conflict) and the Python (has_bidi_conflict, Text.has_bidi_conflict), Node (hasBidiConflict) and Ruby (Disarm.bidi_conflict?) bindings.

  • HostnameAnalysis direction fields (#412). The Python HostnameAnalysis gains bidi_conflict (folded into suspicious), cross_label_script (the broader, non-folded cross-label fact), and label_scripts (per-label resolved scripts, left to right) for position-aware caller policy.

  • Anomaly detection: has_anomalies / inspect_anomalies (#389). An out-of-place-character detector: it flags text disguising a real word via a cross-script homoglyph, leet, single-letter segmentation, a zero-width / bidi control, or zalgo, and reports a technical fact, not intent (like the hostname analysis). Built on the core's own primitives plus a caller-supplied common-word lexicon (used only by the leet/segmentation branches; the others are script-agnostic). Exposed across the Rust core (disarm::api) and the Python, Ruby, and Node bindings, with a per-language usage page. A dated defensive publication — published as prior art so the method stays freely usable.

  • Reusable anomaly lexicon handle (Lexicon). The binding has_anomalies / inspect_anomalies functions rebuilt a hash set from the caller's word list on every call; a new opaque Lexicon class lets callers build the set once and reuse it across many calls (disarm.Lexicon(words) in Python, new Lexicon(words) in Node, Disarm::Lexicon.new(words) in Ruby). Both functions accept either the raw word collection (unchanged, back-compatible) or a Lexicon. The Rust core already amortizes this (it takes &HashSet<String>), so this closes the gap only the FFI bindings had.

  • Node.js docs + doc-example gate (#44). A docs/node/ getting-started page and API reference plug into the language-neutral structure (#50), with Node.js added to the Getting started and API Reference nav. Every Node // => example is executed against the built addon by scripts/check_doc_node_examples.mjs — the Node analogue of the Sybil/Rust/Ruby doc gates — wired into the node CI job (which now also triggers on docs/**), so the examples can't rot.

  • Node.js binding (#44). A new bindings/node/ napi-rs addon exposes the pure-Rust core to Node with a fully-typed, idiomatic TypeScript surface — camelCase functions, options objects with sensible defaults, string-union token types, and a DisarmError / DisarmInvalidArgument class hierarchy. It covers the full plain-function surface (transliterate, confusables, slugify, normalization, text cleaning, graphemes, filenames, reverse/untranslatable, script analysis) and ships .d.ts types. Two layers, like the gem: a raw napi shim (src/lib.rs) under a hand-written index.ts. Built + vitest-tested in CI against the in-repo core (the #374 drift gate, now node/"Node checks passed"), with a publish-node.yml release workflow (per-platform prebuilds + npm provenance) so npm i disarm needs no Rust toolchain.

  • Ruby: filename, reverse-transliteration, and script-analysis ops (#375). Completes the plain-function parity backfill: sanitize_filename (platform:/max_length:/preserve_extension:), reverse_transliterate(lang:) (:el/:ru/:uk), find_untranslatable (→ { char:, offset: } hashes), detect_scripts, mixed_script?, and inspect_auto_lang (→ a :script/:chosen_lang/:reason/:discriminators_hit hash) — thin wrappers over the core disarm::api.

  • Ruby: grapheme-cluster operations (#375). The binding gains grapheme_len, grapheme_split, grapheme_truncate, grapheme_width, and terminal_width — user-perceived-character counting/splitting/truncation and East Asian Width display measurement (ambiguous_wide: false by default), thin wrappers over the core disarm::api. Continues the Ruby↔core parity backfill (#375) and unblocks the graphemes Ruby docs.

  • Ruby: normalization + text-cleaning primitives (#375). The binding gains normalize / normalized? (NFC/NFD/NFKC/NFKD), collapse_whitespace, strip_control_chars, strip_zero_width_chars, strip_bidi, and strip_zalgo / zalgo? — the first batch of the Ruby↔core parity backfill (#375), which unblocks honest normalization/text-cleaning Ruby docs. Each is a thin keyword-argument wrapper over the core disarm::api, carrying the core's defaults (normalize(form: :nfc), strip_zalgo(max_marks: 2), zalgo?(threshold: 3)).

  • CI: the Ruby binding is built and RSpec'd against the local core on every PR (#374). A new ruby job in ci.yml compiles the gem (Ruby 3.1–3.3) and runs rake spec against the in-repo core — not the published one — on any PR that touches the binding or the core it wraps. It injects a CI-only [patch.crates-io] redirect so an unreleased core API change is actually exercised; the registry-core build in publish-ruby.yml is unchanged. A core change that breaks the gem (like the 0.10 tuple→struct return that shipped a broken gem, #364–#367) now fails the new "Ruby checks passed" gate on the PR that introduces it, not silently at release.

  • CI: the docs' Rust and Ruby usage examples are now executed gates (#50). The per-language usage tabs are no longer illustrative — each is run in CI, the way the Python tabs already are (Sybil). scripts/check_doc_rust_examples.py extracts every ``rust doc block, compiles and runs it against the pure core with#![deny(unused_must_use)](so an example that discards its result fails);scripts/check_doc_ruby_examples.rbevals every Ruby# =>line against the freshly-built gem. The Rust gate runs in theDoc testsjob; the Ruby gate runs in the Ruby workflow, now also triggered ondocs/**`. Catches the signature/output drift that the tabs introduced (which had shipped as non-compiling Rust until this gate).

  • Ruby: transliterate now accepts a lang: language profile. Previously the Ruby binding's transliterate exposed only scheme:, so it could not reach the core's per-language profiles (a parity gap vs Python/Rust). lang: accepts a String or Symbol and composes with scheme: — e.g. Disarm.transliterate("Київ", lang: :uk) # => "Kyiv". Implemented over the core's Transliterate builder via a generalized _transliterate_opts shim.

Changed

  • Re-point Greek small letter iota U+03B9 to the i-class, reverting #343 (#436). #343 had re-pointed the bare iota from i/і to the l/vertical-bar class (l/ӏ) to unify {ι, ӏ, ا}. That split the iota family — the accented iotas U+03AF (ί) and U+03CA (ϊ) still folded to i in the same table — and was shadowed in the security presets: security_clean and strip_obfuscation run NFKC first, which decomposes the accented iotas to bare iota, so under #343 the whole family folded to l there. It also contradicted the upstream Unicode TR39 mapping (03B9 → 0069, i.e. i) and missed the dominant spoof — normalize_confusables("bιtcoin") returned "bltcoin" instead of colliding with "bitcoin". The bare iota now folds to i (latin)/і (cyrillic), consistent with its accented forms, so the entire iota family folds to the i-class under normalize_confusables and the NFKC-first presets, and the ι-for-i spoof is caught (bιtcoin → bitcoin). The genuine full-height bars — ӏ (palochka U+04CF), ا (alef U+0627), and the U+2502/U+FFE8 bars (#245) — stay in the l-class. The only confusable-table change is the single iota row in each target.

  • security_clean and normalize_user_input no longer neutralize path separators (#431, reverses #248). The presets previously rewrote / and \ to _ and collapsed .. runs so the output was safe to drop into a filesystem path. That is sink-specific output sanitization — out of scope for the canonicalization presets per THREAT_MODEL.md — and it corrupted legitimate input: URLs, file paths, and any /- or \-bearing string came back mangled ("https://example.com/path" → "https:__example.com_path"). The presets now pass separators through verbatim. Upgrading: if you fed preset output straight into a filesystem path, defend traversal at the sink instead — call sanitize_filename on the final path component, or validate against your own allowlist. A confusable fraction/division slash that NFKC folds to a real / is still normalized to / (that is canonicalization working as intended); it is just no longer rewritten away. The internal neutralize_path_separators helper is removed.

  • collapse_whitespace folds the full whitespace set and the blank-rendering code points; control/zero-width stripping is now a separate step (#433). collapse_whitespace was category-driven and also deleted controls and zero-width characters inline. It now folds whitespace only, to a single space, over an explicit core-defined set: the line controls (TAB/LF/VT/FF/CR), the information separators (U+001C–U+001F), NEL, the Zs/Zl/Zp spaces, and a blank-rendering set that category detection cannot reach — U+2800 Braille blank and the Hangul fillers U+115F/U+1160/U+3164/U+FFA0 (e.g. aㅤb → a b). Breaking: collapse_whitespace drops its strip_control / strip_zero_width parameters (Rust, Python, Node, Ruby) — it no longer deletes anything. Compose strip_control_chars / strip_zero_width_chars before it for the old behaviour; the presets do this internally, so their output is unchanged except for the line-control fix below. strip_control_chars now preserves the whitespace controls (CR/VT/FF/NEL/U+001C–U+001F) so the fold can turn them into a space; it still removes NUL, DEL, and the rest of the C0/C1 block. The PRESETS metadata now lists the explicit strip_control / strip_zero_width steps.

  • security_clean now caps combining marks (anti-zalgo, #429). The preset left zalgo-stacked tokens intact, so a mark-stacked admin did not match its base form in a denylist/dedup comparison security_clean is meant to canonicalize. It now caps combining marks at 2 per base (the same threshold normalize_user_input already used), removing abusive stacking while preserving legitimate diacritics — security_clean stays accent-preserving (café → café, Việt → Việt; full accent folding remains in search_key/sort_key). The cap runs after the invisible/control strip so a stripped character between marks cannot split a run and hide the count (#121), and idempotency is verified by the raw-equality property test. Output change: inputs with more than two stacked marks per base are now capped.

  • is_suspicious_hostname and has_anomalies now flag bidi-direction conflicts (#412). These detectors strengthen as disarm grows. A hostname that mixes strong-LTR and strong-RTL characters (the "BiDi Swap" shape, e.g. varonis.com.ו.קום) is now flagged suspicious via the new bidi_conflict signal — previously it slipped past mixed_script (which is per-label) and was only caught incidentally, if at all. The anomaly detector gains a bidi_mixed finding kind for a token mixing strong-LTR and strong-RTL letters: it is the precise, reorder-capable subset of mixed_script and additionally catches non-Latin RTL mixes (e.g. Cyrillic+Hebrew) the Latin-anchored mixed_script rule could not see. Behaviour change: some inputs that previously reported mixed_script (Latin+Hebrew/Arabic) now report bidi_mixed, and some that reported clean now flag. bidi_conflict=False / no bidi_mixed is not a safety guarantee.

  • sort_key now preserves base accented characters (#99.1). sort_key is documented as a collation key — accented forms should stay distinct so the accent survives for ordering — but it shared search_key's full transliteration pass, so it ASCII-folded every accent ("Über" → "uber") and produced output identical to search_key. It now transliterates only non-Latin scripts, preserving Latin accents (sort_key("Über") → "über", sort_key("Café") → "café") while still folding Cyrillic/Greek/etc. to a consistent Latin form ("Война и мир" → "voyna i mir"). search_key and catalog_key are unchanged — they still fold accents for exact-match lookup and dedup. A language profile no longer expands an accented Latin letter in a sort key (sort_key("Über", lang="de") is "über", not "ueber"). Output change: persisted sort keys for accented-Latin input will differ from 0.10 and should be regenerated. Applies across the Rust core and the Python, Ruby, and Node bindings.

  • Docs: synced the public XMR benchmark claims to the v2 note (#399). The README, the docs landing page, the adversarial-defense page, and the unidecode-migration guide led with the v1 curated-set headline (XMR = 1.000 on the hand-curated pairs). They now lead with the v2 broad-sample measurement over the 1,314 single-codepoint TR39 sources whose skeleton is a single Latin letter: instance XMR 0.634 / 0.682 (95% CI) with ~95% per-source coverage (stated as a distinct quantity), plus the NFKC (0.103) and TR39-skeleton-oracle (1.000, by construction) baselines, citing the v2 DOI 10.5281/zenodo.20618323. The curated 1.000 is retained only as a labeled sanity check, and the curated set is described correctly (18 hand-curated Cyrillic pairs; the 19 Greek pairs were a separate experiment). CITATION.cff is bumped to 0.11.0 with the note DOI.

  • Docs: Node.js usage tabs across the guide pages (#44). The twelve guide pages that carry Python/Rust/Ruby tabs now also show a runnable Node tab — 38 tabs in all, matching the Ruby coverage. Every Node example is executed against the built addon by the doc gate (scripts/check_doc_node_examples.mjs).

  • Docs: completed the language-neutral restructure (#50). The Adversarial-Text Defense concept page now shows Python/Rust/Ruby usage tabs (no bare Python), and the stale untabbed user-guide/getting-started.md was removed in favour of the per-language getting-started guides (now linked from the index nav). With every published binding carrying install + quickstart + API and mkdocs build --strict clean, all four #50 acceptance criteria are met.

  • Docs: Ruby usage tabs across the guide pages unblocked by the parity backfill (#375/#50). The normalization, text-cleaning, graphemes, filenames, and language-detection guides now show a runnable Ruby tab beside Python and Rust — 17 tabs in all. Every Ruby example is executed against the built gem by the doc gate, so the tabs cannot rot.

  • Docs: language-neutral scaffold — first phase of the docs restructure (#50). Reshaped the documentation IA toward "language-neutral concept core + per-language specifics": a neutral landing headline (no longer "for Python") that routes by ecosystem; per-language Getting started pages under docs/python/, docs/rust/, and docs/ruby/; a shared docs/concepts/which-function.md concept page (lifting the #328 decision table into the neutral layer); and an mkdocs.yml nav reorganized into Getting started / Concepts / Guide / API Reference (Python · Rust) / Architecture / Migration / Reference / Project. Folded six previously orphaned pages into the nav. No library behaviour change; the per-topic concept/usage split and per-language example tabs land in following phases.

  • Docs/metadata: scope transliterate() vs the TR39 confusable functions (#328). The headline identity led with "TR39 confusable analysis", while the most discoverable function, transliterate(), performs the opposite mapping — phonetic BGN/PCGN romanization (Cyrillic р → r), not TR39 visual confusable folding (р → p). Clarified across every entry point with no behaviour change: the identity one-liner (README, docs/index.md, Cargo.toml, pyproject.toml, mkdocs.yml, CITATION.cff) now says visual confusable analysis and phonetic transliteration; a new "Which function do I want?" decision table sits near the top of the README and docs landing page; and transliterate()'s docstring (hence docs/api/transforms.md) and the README Quick Start block now state it is romanization, not homoglyph defense, pointing to normalize_confusables() / strip_obfuscation() for the latter.

Deprecated

  • Presets renamed to mechanism names; old names deprecated (#430). The three presets whose *_clean / normalize_user_input names overpromised safety — flagged as documentation defects in THREAT_MODEL.md — are renamed to names that describe their mechanism. The rename is byte-stable (old(x) == new(x) for all inputs):
Old name (deprecated) New name
security_clean canonicalize
display_clean strip_format
normalize_user_input canonicalize_strict

The old names remain as deprecated aliases across every binding — Rust (free functions + DisarmStr methods, #[deprecated(since = "0.11.0")]), Python (each emits a DeprecationWarning; the Text builder's .security_clean() / .display_clean() methods and the PRESETS keys are aliased too), Node (securityClean, @deprecated), and Ruby (Disarm.security_clean, warns with category: :deprecated). They are removed in 1.0. catalog_key, search_key, sort_key, ml_normalize, and strip_obfuscation are unchanged.

Fixed

  • Hardening-review follow-ups (M-2, M-3, L-1, L-2). A pass over the 2026-06-20 deep review closed four small correctness/perf/security gaps: (M-2) the eager normalize_confusables no longer unconditionally allocates and rebuilds the string on a pure-ASCII / already-folded no-op — it now delegates to the borrowing form, sharing its borrow-on-no-op fast path; (M-3) that borrowing form skips the needs_composition char-decode scan on pure-ASCII input via a cheap is_ascii() short-circuit; (L-1) the Slugifier / UniqueSlugifier default= constructor kwarg — which crosses the str→Rust boundary in __init__, outside the @_surrogate_safe-guarded __call__ — is now WTF-8→UTF-8 scrubbed, so a lone-surrogate default no longer raises UnicodeEncodeError (closing the last gap in the #476 contract); (L-2) the fast-path guard's Confusables step now sets its marks bit itself rather than relying on a preceding Nfkc step, so a hypothetical Confusables-only preset can't let the guard skip a decomposed homoglyph the fold would recover. Per-cluster allocation in compose.rs (L-6) is also removed via a reused NFC scratch buffer, and a stale generator comment claiming U+0344 is "unmapped" is corrected (it maps to the empty string, so its output-neutral row is emitted, not skipped).
  • normalize_confusables is idempotent on an excluded singleton followed by an unrelated mark. The compose-at-lookup pass (#481) recovered a composition-excluded precomposed singleton (ড় U+09DC = ড + nukta) only by a whole-cluster widening-map lookup. When such a singleton was followed by an unrelated combining mark (a visarga), the cluster's .nfc() decomposed the singleton (ড় ◌ঃ → ড nukta visarga) and the trailing mark made the lookup miss the 2-char ড nukta key — so it stayed decomposed. The fold then oscillated (a bare ড় composes, ড় + mark decomposes), i.e. nc(nc(x)) != nc(x), surfaced by a normalize_confusables_idempotent proptest seed. The lookup now matches the widening map by greedy longest prefix at each position in the cluster (bounded by a build-time-emitted EXCLUDED_COMPOSITIONS_MAX_KEY_CHARS), so the excluded head recomposes and any trailing marks are kept — idempotent and form-invariant. The mark-free hot path and the common single-mark cluster are unchanged.
  • sanitize_filename never returns an empty name or a . / .. directory reference; leading/trailing dot hygiene (#485, #487). Three correctness gaps with one root: the extension branch re-prepended '.' and was exempt from the stem's dot trim. (1) The empty string bypassed the never-empty fallback and returned "" — os.path.join(dir, "") targets the directory, a write-target footgun. (2) Trailing dots/spaces survived ("report..." -> "report.", "CON." -> "_CON."), which Windows then silently strips at the filesystem layer. (3) A separator-then-dot-like input reduced to a bare "." ("_" + U+00B7 -> "."), a current-directory reference. A shared finalize_name now runs on the fully assembled name: it trims leading and trailing dots and spaces, and falls back to "_" for an empty, ".", or ".." result — so the output is always non-empty, never a directory reference, and never a leading/trailing-dot dotfile, across both return paths. A 50-case attacker battery (path traversal, Unicode separator homoglyphs, control/NUL, RTLO/bidi, the ADS colon, dot hygiene, the separator-plus-dot class) and non-emptiness/idempotency property tests lock the defenses in. (A separate idempotency gap from the extension-split boundary moving between passes is tracked in #488.)

  • Class-based entrypoints honor the malformed-Unicode (surrogate) contract (#476). The #469 boundary adapter wrapped every module-level _core callable, but the class entrypoints that cross the str → Rust boundary on construction or in a method were not covered: Lexicon(["a\ud83d…"]) raised UnicodeEncodeError, and so did calling a Slugifier / UniqueSlugifier / TextPipeline on surrogate-laced text. They now apply the same WTF-8 → UTF-8 scrub-and-retry: the three callable classes guard their __call__, and Lexicon (a frozen, non-subclassable PyO3 class) is guarded by a metaclass proxy whose construction scrubs while isinstance still recognizes every real handle, so the has_anomalies / inspect_anomalies prebuilt-handle dispatch is unaffected. The dynamic surrogate audit is extended to enumerate the exported classes (covered or reviewed-exempt), so a future class with a text surface fails the audit rather than silently skipping the contract.

  • Hangul romanization is invariant to the input's normal form (#483). A precomposed syllable run was romanized with inter-syllable spaces (처리 → "cheo ri"), but the same text decomposed to conjoining jamo (NFD) romanized contiguously ("cheori"), so the output depended on the normal form. Conjoining jamo are General_Category=Lo, so #479's General_Category=Mark compose-at-lookup gate never fired on them. The fix composes an L + V [+ T] jamo run into its syllable by the standard Unicode index arithmetic (no table, no normalization pass), gated on a cheap jamo range check, as a sibling to the existing mark-composition path — so the decomposed form takes the same per-code-point path as the precomposed one. transliterate, slugify, unidecode, and slugify_unicode now agree across NFC/NFD/NFKD on Hangul; the precomposed output ("cheo ri") is unchanged. Partial jamo (a lone L, or L + T with no vowel) are left alone. Cosmetic spacing only — both forms always recovered the same Korean reading; this was the last NFC/NFD gap on the transliterate path.

  • Close the raw-vs-normalized residual the #477 oracle could not see (#481). The form-invariance audit compared the normal forms against each other but never against the raw precomposed input, so a composition-excluded code point passed green while still degrading: Devanagari क़ U+0958 transliterated "qa" raw, but its canonical decomposition KA + nukta is composition-excluded, so every normal form degraded to "ka" (the mark dropped). Closed entirely with build-time data, no runtime canonicalization pass (so the #478 decompose-then-recompose regression class cannot recur): an exclusion-inclusive compose map widens #479's compose-at-lookup so a base+mark exclusion reaches its precomposed scalar (KA+nukta → QA, shin+sin-dot → שׂ U+FB2B, Tibetan vowel stacks, the Hebrew presentation forms), and the two real Greek-oxia confusable singletons (U+1F77/U+1F79 → i/o) become char-table rows. The map is gated on the precomposed target being mapped, which keeps an unmapped operator (FORKING U+2ADC = NONFORKING + U+0338) from cycling with the NFKC recovery. The audit now asserts f(raw) == f(NFC) == f(NFD) == f(NFKD) for the transliterate family and confusable detection, with a small characterized tail (two Greek accent-punctuation code points, and the benign spoof-resolutions where normalizing a look-alike to its genuine character — Kelvin U+212A → K — flips detection). The ~1,027 non-target singletons (ά U+1F71 vs U+03AC, the same Greek letter, neither a Latin confusable) are deliberately left as benign re-encoding: normalize_confusables is a targeted fold, not a normalizer.

  • Confusable folding, detection, and transliteration are invariant to the input's normal form (#475, #477). The confusables maps and the transliteration tables are keyed per code point on the precomposed form (ї U+0457 → i / yi), so a decomposed input (і U+0456 + combining diaeresis U+0308) reached only the base entry and the mark survived — an attacker could evade the recovery, or flip is_confusable, just by sending NFD. The confusables fold and detect (normalize_confusables, is_confusable) and every public str → str recovery entrypoint — transliterate, unidecode, and the whole slugify* family (including the Unicode-preserving slugify_unicode) — now compose each base + combining-mark cluster at lookup time, so the result is invariant to the input's normal form (f(NFC(x)) == f(NFD(x)) == f(NFKD(x)), and likewise for the is_confusable predicate). The composition is compose-only (it never decomposes): a composition-excluded presentation form such as Hebrew שׂ U+FB2B keeps its own table entry (→ s), where a naïve "NFC the input first" would have decomposed it and changed the output. It is gated on a combining-mark check, so mark-free input (ASCII, CJK, precomposed letters) keeps its borrow/zero-allocation fast path; it composes the full cluster (Vietnamese ệ, polytonic Greek ᾷ, and Brahmic two-part vowels like Bengali ো). A self-guarding audit enumerates the public entrypoints and asserts the invariant, so a future entrypoint that forgets to normalize fails the test, not in production.

  • Digit confusables fold to their digit, not a look-alike letter (#439). The confusable maps mapped many non-ASCII digit sources to letters or punctuation — Arabic-Indic ٠→., ١→l, ٥→o, Devanagari/Bengali/NKO zeros→o/O, and the Unicode 16 outlined digits 𜳰→O / 𜳱→l. The root cause: gen_confusables.py classifies digits via unicodedata, so running it under a Python whose Unicode table is older than the bundled confusables.txt silently mis-folds any digit that table doesn't yet know. The generator now (a) folds every Nd digit source to its canonical ASCII digit and (b) refuses to run under a Unicode table older than the data (warning on any mismatch). The maps are regenerated: every digit spoof now canonicalizes to the plain digit (٠/०/𜳰 → 0), keeping numbers numeric (the llm_guardrail "digits are never remapped to letters" guarantee).

  • sort_key / search_key / catalog_key are now idempotent across scripts and cases (#419). The transliterating key presets ran transliterate before fold_case, so a cased letter whose folded form is in the table but whose original is not — e.g. a Georgian Mtavruli capital Ჱ (U+1CB1), absent from the table, folds to Mkhedruli ჱ which transliterates to he — only transliterated on the second pass, violating f(f(x)) == f(x). fold_case now runs before transliterate so both passes see the same form. search_key/catalog_key additionally fold again after transliterate, since full transliteration can emit uppercase ASCII (£→GBP, №→No) that the pre-fold can't reach — those keys are now lowercase and stable. Output change: a few currency/symbol inputs that previously produced uppercase keys now fold to lowercase. Idempotency is pinned by per-preset property tests.

  • security_clean / normalize_user_input idempotency on duplicate combining marks (#434, #416 residual). A duplicate combining mark broke the single NFC → confusables → NFC sandwich: NFC composed only one mark onto the base, the TR39 fold dropped it, and the recomposing NFC reattached the spare mark — re-creating a foldable composed character the next call would consume, so f(f(x)) != f(x) ("c"+◌̧+◌̧ → "ç" then "c"). The confusable fold is now iterated to a fixed point (each pass removes ≥1 mark, so it converges in a couple of iterations), making both presets true fixed points. The #416 Hypothesis idempotency property is re-broadened and the normalize_user_input Rust proptest strengthened from nfc-modulo to raw equality.

  • Line controls no longer join tokens in collapse_whitespace (#433). TAB and LF folded to a space, but VT, FF, CR, NEL, and the information separators (U+001C–U+001F) were deleted — so a + CR + b became ab while a + LF + b became a b. All of them are Unicode whitespace; deleting them was an invisible-join (coalescence) vector. They now all fold to a single space, so a\rb → a b. The blank-rendering Braille and Hangul fillers, which category detection passed straight through, are folded too.

  • security_clean / sort_key idempotency on invisible-separated combining marks (#416). When an invisible code point separated a base character from a combining mark (e.g. "a" + U+200B + combining acute + "b"), the leading NFKC passed over the still-separated mark and the later zero-width strip then left the base and mark adjacent but decomposed — so the composed form appeared only on the second call, violating the documented f(f(x)) == f(x) invariant (which THREAT_MODEL.md classifies as a vulnerability). An NFC pass after the strips now recomposes the adjacency on the first call, in the Rust core, so every binding inherits it. For security_clean a second, deeper cause was also fixed: TR39 confusable skeletoning is not normalization-stable (it drops the diacritic on some composed accented letters — ç→c, ø→o — but not the decomposed form, and can emit a decomposed skeleton like Ý→Y+◌́), so the confusable fold is now sandwiched between two NFC passes and the pipeline is a verified fixed point under a strengthened raw-equality proptest. Output change: for these previously non-idempotent inputs the first call now returns the composed NFC form. sort_key was affected only because it began preserving accents in #411 (search_key/catalog_key, which fold accents away, were never affected). A separate, pre-existing sort_key non-idempotency (transliterate-before-fold-case on a case pair) is tracked in #419.

Internal

  • Dependency-freshness audit across every manifest + full dependabot coverage. Dependabot only watched the root cargo/uv/actions manifests, so the binding crates rotted a full major unseen (napi 2→3, magnus 0.7→0.8). .github/dependabot.yml now watches every manifest — the core crate and both binding workspaces (cargo), the Node package (npm), and the Ruby bundle (bundler) — and a new dev-time scripts/audit_dependencies.py audits all of them against their registries in one command (--strict to fail on a major lag), run weekly by the dependency-audit workflow. The guard makes any future config gap visible instead of silent. See DEPENDENCY_UPGRADES.md. The DCO check now exempts trusted GitHub App bots (*[bot] authors, e.g. dependabot[bot]) — matching the official DCO app's default — so dependabot's PRs can finally satisfy branch protection and auto-merge instead of every bump being silently blocked.

  • The Tier 3 exhaustive+formal gate now guards every publish, not just PyPI/crates.io (#159, #395). The pre-publish regimen — the exhaustive Rust domain tests (#[ignore]) and the Python formal invariants (@pytest.mark.formal) — moved out of an inline job in publish.yml into a reusable workflow_call workflow (.github/workflows/tier3.yml) that all four publish paths depend on: the PyPI wheel, the crates.io core, the RubyGems gem, and the npm addon. Previously only the wheel and the core were gated, so a release whose core failed the exhaustive net could still ship the bindings. Also wired the exhaustive grapheme-integrity suite (exhaustive_grapheme, #174) into the gate alongside exhaustive_transliterate — it was documented "run before release" but had never actually been in the release workflow.

  • Binding publish workflows build against the in-repo core on non-publish events (#374, #396). publish-ruby.yml's test job and publish-node.yml's build job compiled the binding against the published core, so a pre-release binding that calls a core API not yet on crates.io (e.g. has_anomalies before this release) failed to build on every PR/push — red on main until the matching core shipped. They now apply the same CI-only [patch.crates-io] redirect to the in-repo core that ci.yml's drift gate uses, but only on push / pull_request; on release / workflow_dispatch the shipped gem and prebuilt addon still build against the published core, unchanged.

  • Node binding: bumped vitest 3 → 4, dropping a vulnerable dev-only esbuild (#392, #394). The Node binding's test runner pulled in esbuild 0.27.7 — a dev-only transitive dependency, never part of the published npm package — which carried two HIGH advisories (GHSA-gv7w-rqvm-qjhr, GHSA-g7r4-m6w7-qqqr). vitest 4 pulls vite 8, which demotes esbuild to an optional peer dependency, so the vulnerable package drops out of the resolved tree entirely (npm audit reports zero vulnerabilities). The Node test matrix is unchanged (20/22).