Changelog: 0.14.x¶

Archived verbatim from CHANGELOG.md. Newer releases are in the changelog; the archive index lists every older series.

[0.14.1] — 2026-08-28¶

Added¶

Every documentation page now says which commit it was built from, and which version you can install (#641). The site deploys from main on every push; the package comes from the newest tag. At the worst point those were 68 commits apart, and docs/security/cve-validation.md named five entry points that raised AttributeError on the release it was describing — is_case_fold_stable, find_key_collisions, unmapped_confusables, find_unmapped_confusables and CONFUSABLES_VERSION.

A mkdocs hook (scripts/mkdocs_build_banner.py) stamps the banner onto all 91 pages. The version comes from PyPI when the docs workflow can reach it and from pyproject.toml otherwise, so a local mkdocs serve shows the same banner a deploy does.

A scheduled job that checks the documentation against the published wheel (#641). scripts/check_docs_against_release.py collects every disarm name the docs use (imports and attribute access in Python blocks, disarm.x mkdocstrings directives, inline code spans in prose) and fails when one of them does not exist in the installed package. Run against 0.13.0 it reproduces all five names above; against 0.14.0 it is clean.

It runs weekly rather than on pull requests, because documentation ships with the feature it documents and a PR gate would block every feature branch for doing that correctly. A red run means documented API has outrun the last release.

Names it cannot fix are allowlisted, and only against an open issue. The gate fails when a listed name starts resolving, and a test fails when the page that named it is gone, so the list shrinks rather than accumulating — it did so in this same release. What remains is LANG_AUTO, the one LANG_* constant of 84 that the package never re-exports even though three doc blocks tell readers to import it (#660, found by this gate on its first run).

Nothing checked that an installed artifact imports and runs; now two jobs do (#667, #669). Every existing gate tested a development environment. CI built a wheel, installed it, then installed the project again from source with its test extras on top, so what pytest imported afterwards was not reliably the wheel that was built and the wheel was never imported alone. The local pre-push gate uses maturin develop, which produces no distributable artifact at all.

scripts/smoke_installed.py is the shared body: import disarm, assert __version__ is not the "0.0.0+unknown" missing-metadata fallback, call one function per public surface, and pin the one documented path that raises on every released artifact (transliterate(..., context=True), whose message names bootstrap_dicts.sh). It imports nothing outside the standard library, so it runs in a virtualenv holding the artifact and nothing else — the environment in which a missing runtime dependency is detectable at all. It also refuses to pass when a source tree shadows the install, which is how such a check silently tests nothing.

.github/workflows/smoke.yml runs it twice. The tracked-tree job exports git archive HEAD, installs it and runs the smoke body, on every push to main with no paths filter — ci.yml triggers on pull_request only, so the commit that lands was the one commit nothing built. The artifacts job builds a wheel and an sdist and installs each into a clean virtualenv; the sdist had no coverage anywhere, having been built at publish time and never installed.

Verified locally before landing, against real installs rather than in principle: the sdist builds from source and passes all 14 checks, and so does the tracked tree. Both were sound, so the gates encode a true invariant rather than papering over a break.

Key-builder output is gated, not just promised (#644). 0.14.0 stated the contract — a patch release never changes search_key, catalog_key or sort_key output; a minor release may — and said plainly that nothing enforced it. tests/test_key_stability.py does now: eight key-producing functions recomputed over a fixed 22,878-row corpus, failing with a per-function count and a sample of what moved.

Review was not enough, and the history is the argument. 0.14.0 moved search_key on 4.1% of a 5,030-input corpus, and the change responsible (#602) was a correctness fix whose diff said nothing about keys — it stopped ErrorMode::Preserve excepting itself from the table's empty mappings. Nobody reading that diff would have thought reindex.

Checked against the published 0.13.0 wheel the gate reproduces exactly that movement: sort_key 3,026 rows of 22,878 (13.23%), canonicalize_strict 604, search_key and catalog_key 267 each, down to strip_obfuscation at 164. Every one is a correctness fix, which is the point rather than a complication — banĸ.example really should become bank.example, and it still invalidates a stored key. The gate does not judge whether a change is right; it makes the change visible and forces somebody to decide.

Two properties of the fixture are recorded rather than hidden, in tests/fixtures/key_stability/README.md. The corpus is not reproducible — its natural rows are tokenised from randomly sampled Wikipedia titles, so the committed file is the fixture, while the derived keys regenerate deterministically from it. And its licence is not the repository's: corpus.txt is CC BY-SA 4.0 with attribution, where the rest of disarm is MIT.

What is still missing is a signal a consumer can assert in their own CI. That is KEY_SCHEMA_VERSION (#645), and it is downstream of this: a constant only means something once something detects that the thing it counts has moved.

An execute-only doc tier, so no page has a runnable example that nothing runs (#656). Before it, a page could be in three states: on EXECUTED_RECIPES with its assertions checked, carrying no python blocks at all, or — the invisible one — carrying blocks that nothing executed. Eight pages were in the third state, so a signature change could break a published example in silence.

EXECUTE_ONLY_RECIPES runs those blocks and checks nothing else. That claims less than the assertion list on purpose, and leaves the ratchet exactly as it was: a page still joins EXECUTED_RECIPES only once its examples assert rather than decorate.

Getting the seven pages to pass took five different fixes, because the blocks failed for five different reasons. Three fragments on limitations.md needed the imports the page never made; architecture/pipeline.md needed a dataset to iterate. Three genuinely cannot run and now say so: api/index.md uses mypy's reveal_type, api/encoding.md ends on a call it documents as raising, and migration/index.md imports a comparator from the bench extra.

tests/test_doc_recipe_coverage.py keeps the third state gone, which the tier cannot do for itself — a new page joins the tree without touching either list. The doc-test runner now covers 39 pages, up from 32.

Fixed¶

Four CI workflows built a dev-profile wheel and then tested it (#658). maturin build defaults to the dev profile. --release was missing from ci.yml's test and doc-tests jobs, from tier3.yml's formal-invariant build, and from bench.yml, while nightly-hypothesis.yml and perf-gate.yml already passed it — so the convention existed and these four sat outside it.

Measured on one machine with one selection: the CI Python job takes 129s against a debug wheel and 17s against a release one. The extra compile time is repaid several times over by the test run it feeds.

bench.yml was wrong rather than merely slow. It built a debug wheel and ran benchmarks/bench_quick.py against it, so the numbers that job printed described an unoptimized build. A benchmark whose output cannot be compared to anything is not a smoke test of the artifact the project ships.

No published artifact was affected: publish.yml has always passed --release to maturin-action, so every wheel on PyPI is an optimized build.

Six CVE rows report and the matrix showed a dash (#665). docs/security/cve-validation.md is generated from a registry, gated against it by TestDocsMatrixDrift, and understated disarm's own detection on six of its 46 rows. The gate was comparing the page with the registry; the registry was what was wrong.

The root cause is one missing dictionary entry. DISPOSITION_LABELS had a label for {not-affected, detected} and none for {out-of-scope, detected}, so adding DETECTED to an out-of-scope row raised KeyError and the combination could not be written down at all. Four rows gained their signal with compat_fold (#633); two — CVE-2026-28289's leading zero-width space and CVE-2024-3098's fullwidth __import__ — reported before that and had never been recorded.

A carve-out in test_detectors_are_exactly_those_that_fire skipped out-of-scope rows entirely, on the reasoning that a predicate might fire incidentally and noticing a character is not defending a CVE. Measured against the registry, no such row exists: all six fire on the mechanism the CVE exploits — the fullwidth solidus that is the path bypass, the zero-width space that is the upload bypass. The carve-out was suppressing six real signals to guard a case that does not occur, so every row is compared now.

Two published counts that did not add up (#665). 25 + 14 misses 46, and 15 + 2 + 1 misses 14; the correct census is 25 compared, 15 out of scope, 5 not affected, 1 detected-only. That sentence had been wrong since before 0.14.0. The most recent detection total a reader met was also stale: has_anomalies reports 24 of the 46 rows, not the 19 it reached after #612.

Both numbers are now derived checks rather than prose. TestDocsMatrixDrift parses the sentences and compares them with the registry, and requires the arithmetic to reconcile with itself so a future edit cannot make each number individually right and the sum wrong.

Eight detector claims were verified by nothing (#665). DETECTOR_PANEL covers 33 of the 41 claims on the page. The rest are surface-specific — is_suspicious_hostname, inspect_anomalies, is_case_fold_stable — and were checked only by hand.

SURFACE_DETECTORS now records what firing means for each, because they do not all report the same way: is_suspicious_hostname returns a tuple, and is_case_fold_stable signals a problem by returning False. They are checked in one direction only, since running a hostname predicate over a source-code probe would manufacture coverage. A companion test fails if a claim ever names something neither table knows how to check.

docs/index.md had drifted from the README it is generated from, in both directions (#656). The file carries a "do not edit directly" banner and is produced by scripts/generate_docs_index.sh from README.md + docs/_index_nav.md. Nothing ran the generator and nothing checked its output, so the banner was the only thing holding the line.

Two Features bullets existed only in the generated file, where the next run would have deleted them. A Node.js Getting Started entry, a whole-script-spoof example and a coverage-residue note existed only in the sources and had never reached the published site. The second kind is the one that reads as working: the change appears on GitHub, and the docs site quietly does not move.

Both reconciled — the orphaned bullets moved into README.md, everything else regenerated — and scripts/generate_docs_index.sh --check now fails when the three files disagree. It runs in CI's doc-tests job rather than test, because test is gated on a path filter that does not include **.md, so a README-only pull request would have skipped it.

This is also what executes the README. All ten of its python blocks land in docs/index.md, which is first on EXECUTED_RECIPES and runs under Sybil on every CI run. In sync, the most-read file in the project has its examples asserted; out of sync, it does not. tests/test_docs_index_drift.py pins both halves, including that the check can fail and that the generator is idempotent.

README.md opened with the wrong function (#656). Its first runnable block reached for strip_obfuscation, normalize_confusables and is_suspicious_hostname — three narrow tools — while most readers arrive wanting to clean untrusted input, which is canonicalize. That name appeared in no runnable block anywhere near the top.

canonicalize now leads the block and the Which function do I want? table, with the specialists below it. Being in a generated, executed page, both new lines are asserted rather than decorative.

The test suite spent most of its time on two tests and a serial loop (#658). Item 1 landed earlier — --release on four maturin build calls, which took CI's Python job from 129s to 17s. The rest of the issue is here.

TestBatchReleasesGil was 112.6s of that 129s: 87% of the job in two tests. Both assert a ratio, so the batch only has to be large enough that per-call overhead does not blur it. Measured across sizes, the speedup is flat at ~1.8x from 21.6M characters down to 0.5M, against a 1.3x threshold. Sized to 4.3M with rounds cut from 5 to 3 — deliberately not the smallest that works, because CI runners have fewer cores and noisier neighbours than the machine this was measured on. The pair now takes 0.39s, and ran ten times without a flake.

scripts/run_doc_tests.py ran 32 pytest processes in sequence for about 4.6s of actual assertions. The pages are independent — that is why they get separate processes — so the loop is concurrent now: 6.5s to 1.7s. Output is buffered and only failures print theirs, in allowlist order, so a concurrent run reads like a serial one. DISARM_DOC_TEST_JOBS=1 restores the serial behaviour.

The oracle suite emitted 42,457 DeprecationWarnings on a full run, from exercising the three deprecated aliases against every generated input. Filtered by name in that module — not globally, which would also hide deprecations from dependencies, the ones a maintainer wants to see.

Bare pytest now runs what CI runs (#658). The Hypothesis tier was in the local default and in no CI job, so a contributor paid ~67s per run for a tier nightly-hypothesis.yml already exercises every night with a random seed and a 10× oracle budget. And the slow marker described itself as deselectable while nothing deselected it, so it had no effect: both things it covers are gated elsewhere, and one costs a cold cargo build on the first run after a Rust change.

Both are one command away — pytest -m hypothesis, pytest -m slow — and pytest-xdist is in the test extra for pytest -n 2 --dist loadfile. --dist loadfile is not optional: register_lang mutates process-global state that cannot be undone, so tests must stay grouped by file.

CI keeps its serial command. Measured after the fixes above: serial 6.1s, -n 2 4.9s, -n auto 5.4s — auto is worse, because once the suite is short enough, worker startup dominates. The ~1s does not pay for installing the plugin.

CONTRIBUTING.md's tier figures were stale in both directions and are now measured: ~1,025 Rust tests rather than ~630, ~4,490 Python rather than ~2,200, and the Hypothesis tier 587 tests / ~67s rather than "~440 / ~40s".

Three exhaustive test targets moved from release-time to PR CI, and one of them had never run at all (#658 item 7). exhaustive_transliterate, exhaustive_grapheme and width_conformance cover the full BMP, every Hangul syllable, all CJK ideographs, 15 Indic blocks, grapheme-boundary integrity, and every Unicode scalar for width bounds. They ran only in tier3.yml, which publish.yml calls — so a regression in any of it surfaced on the release pull request rather than on the one that caused it, and the reason recorded for that placement was cost.

The cost is 0.62s. The test job already builds the debug profile these binaries need, so nothing is compiled twice. (Release would run them in 0.07s and need a whole second build profile, which costs far more than the 0.55s it saves.)

width_conformance was worse than that: it appeared in no workflow and in no documented gate, so nothing had ever run it. exhaustive_confusables stays in tier3.yml, where its 1.14s and its need for the release profile belong.

test_cli.py spawned 58 interpreters to test argument parsing (#658 item 6). It was 2.73s of a 6.16s suite, and roughly 4.6s of the 5.2s originally measured was interpreter startup. Every test routed through one run_cli helper, so the conversion is in the helper: it now patches sys.argv, sys.stdin, sys.stdout and sys.stderr, calls main(), and turns SystemExit into a returncode. No test body changed.

58 tests, 2.73s → 0.08s.

Four subprocesses remain, in TestProcessEntryPoint, and they are the reason this is a split rather than a wholesale conversion. An in-process suite passes just as happily when python -m disarm no longer resolves, when __main__.py fails to import under a fresh interpreter, or when the console-script entry point is wrong. Those four assert exactly that, and nothing else.

Verified the conversion did not hollow the tests out: planting a RuntimeError in cmd_transliterate turns 34 of the 62 red, across both paths.

fuzz/ deleted, and SECURITY.md no longer claims something the code did not support (#679). The four cargo-fuzz targets have not compiled since the June presets rename: every one imports _disarm, and the crate is disarm. Three also reach pub(crate) modules that an external crate cannot see. Confirmed with cargo check — error[E0433] on all four.

Nothing was going to notice. There is no [workspace] in the root manifest, so no root-level cargo check, clippy --all-targets or test reaches the directory, and no workflow or script invokes cargo fuzz.

The part that made this more than housekeeping: SECURITY.md told vulnerability reporters the library "is exhaustively fuzzed", in the paragraph that sets the bar for a report. It now names what actually runs — 166 proptest properties on every pull request, 23 Hypothesis fuzz tests over arbitrary text and bytes nightly, and 19 falsifying examples pinned across 6 committed regression corpora. That is a stronger claim than the old one, and unlike it, one a reader who checks will find.

THREAT_MODEL.md's "fuzzed and tested for no-panic and linear behavior on hostile bytes (#78)" needed no change: it refers to tests/test_encoding_fuzz.py, which runs.

Documentation¶

Key-builder output now carries a stated stability contract (#644). 0.14.0's Upgrade notes said "Whether their output carries a stability guarantee is open (#644)" after a release that moved search_key, catalog_key and sort_key. The answer, now written into docs/RUST_API.md, RELEASING.md and the three docstrings on both the Python and Rust surfaces:

A patch release never changes key-builder output. A minor release may.

docs/RUST_API.md's data-driven-output clause named normalize_confusables, strip_obfuscation, is_suspicious_hostname and the canonicalize* presets, and not the three functions whose entire purpose is to produce a value you store. They are named now, with a Key stability section covering what a consumer does at each upgrade.

The rule is a description rather than a new constraint. Measured across every version disarm has published, on 12,285 fixed inputs (U+0020–U+2FFF plus a word list in 13 scripts), each release installed from PyPI into a clean virtualenv:

transition		`search_key`	`catalog_key`	`sort_key`
`0.9.0` → `0.9.1`	patch	0	0	0
`0.9.1` → `0.10.0`	minor	0	19	0
`0.10.0` → `0.11.0`	minor	62	73	1021
`0.11.0` → `0.11.1`	patch	0	0	0
`0.11.1` → `0.12.0`	minor	0	0	0
`0.12.0` → `0.13.0`	minor	0	0	0
`0.13.0` → `0.14.0`	minor	147	148	416

Both patch releases moved nothing; three of five minors moved nothing either, which is why a consumer could not tell the two apart from outside.

The golden-corpus gate that would enforce this rather than leaving it to review is still open work on #644, and KEY_SCHEMA_VERSION (#645) is the signal a consumer could assert in their own CI. The document says plainly that neither exists yet.

docs/architecture/emoji-plugins.md is deleted; it documented a plugin system that never shipped (#655). The page named five pip packages with sizes, a disarm.emoji module with FileProvider and ChainProvider, and a disarm-emoji-pack CLI. None of it exists, on main or on PyPI, and the page sat in architecture/ alongside descriptive pages with nothing to tell a reader which it was. Its only marker was a  comment, invisible in the rendered page — the one place a reader never looks.

The shipped provider API is unaffected and was already documented elsewhere: EmojiProvider in docs/api/enums.md, set_emoji_provider in docs/api/transforms.md, and the per-call → global → built-in resolution order in docs/architecture/emoji-engine.md. That page gains the deleted one's scope statements (no emojize, no rendering, no sentiment scoring, no platform rendering history, no versioned emoji data) plus a note on the bundled CLDR vintage.

The unbuilt design is preserved in #662 rather than thrown away: versioned emoji provider packages, a file-based provider, and chaining for mixed-era corpora, which is the piece with no workaround today.

Found independently by the drift gate added in #641 on its first run. Deleting the page took its allowlist entries with it, leaving only #660.

The migration guides now say which library to install (#657). All seven import the module they compare against (anyascii, pathvalidate, slugify, unidecode, text_unidecode, confusable_homoglyphs), and pip install disarm brings none of the distributions that provide them. A reader copying a Before block got ModuleNotFoundError naming a package they never asked for, with no way to tell whether disarm was broken. Six are pinned in the bench extra, which the note names; confusable-homoglyphs is in no extra at all, so that page spells out the package.

It never failed in CI because those blocks carry , which is the category of defect that survives a green pipeline.

normalize says which Unicode version it implements (#642). disarm normalizes to UCD 17.0.0, via the unicode-normalization crate. A host unicodedata on an older UCD disagrees for code points assigned in between — swept exhaustively, every scalar value against all four forms, a UCD 16.0.0 host diverges on exactly one code point (U+A7F1, in NFKC and NFKD), and an older CPython on more. Every divergence is disarm being more current rather than wrong, but a pipeline that canonicalizes with one and validates with the other will disagree about which strings are normalized, and the disputed set moves when the deployment's Python is upgraded.

Stated on the Python and Rust docstrings, in docs/provenance.md as a new row, and on docs/security/cve-validation.md beside the differential assertion that motivated it — that assertion holds for its payload and is not true in general. The runtime accessor is still missing; that half is #642 and #645.

All three statements are gated. unicode-normalization is a floating 0.1 requirement, so a cargo update can move the bundled Unicode data with no disarm code change at all, and three prose claims would keep naming the old version. tests/normalization_ucd_drift.rs compares each of them against the crate's own UNICODE_VERSION const and fails when one falls behind.

The CVE page names the digit_policy trade (#646). Its stated purpose is answering which call do I make when I don't know which attack is coming, and it mentioned digit_policy zero times. Under tr39 a Gurmukhi zero standing in for o folds and the spoof is caught; under the default it is missed. The cost is that an Arabic-Indic year loses its zero to a full stop, which in a path-shaped key is a traversal-adjacent shape. Both directions are now asserted examples on an executed page.

It also records that the flag reaches normalize_confusables and nothing else: not the presets, not the key builders, not any of the seven profiles. A caller following that section's advice cannot select it at all.

Corrected language lists and one wrong method name (#628). The tagline said "bindings for Python, Ruby, and more" for a library that ships Rust, Python, Ruby, Node.js, Java, Kotlin and a C ABI, and the Get started in your language row offered three of them. Node.js is added with its existing page; Java and Kotlin get their Maven coordinates and a pointer, since their guide is still to be written.

Two pages read "analyzeHostname in Node/Ruby/Java". Ruby spells it analyze_hostname, and Kotlin was missing. Both corrected.

Nine transforms now say what they do not do, measured rather than asserted (#653). Only three carried a scope warning. The six that did not include ml_normalize, which passes bidi controls, private-use characters and homoglyphs straight through and is picked by name for exactly the job it does not do.

Each new warning states something measured on the published wheel, not a general caution. The sharpest is the key builders': their homoglyph collisions are a side effect of transliteration rather than a confusable fold, so a lookalike whose script romanizes to something other than its lookalike does not collide at all. Cherokee Ꮃ looks like W and romanizes to la, so Ꮃorld keys as laorld and never meets world — in catalog_key too, whose confusable step runs after transliteration and never sees the character.

One correction to the issue: strip_obfuscation was counted among the six that do not warn. It is the best-documented function of the nine; its statements were bold paragraphs rather than admonitions, which is what the survey was detecting. Its markup-safety paragraph is promoted to a Warning: so it renders as one.

has_anomalies at the seam is documented (#653). Running it on the output of a transform reports whether that transform left something behind, needs no new API, and appeared nowhere in the docs. docs/user-guide/anomaly-detection.md now shows it with asserted examples, and states the caveat that makes it usable: a true result means you chose the wrong function, a false result means nothing. At the reported 42.6% recall it is a useful alarm and a useless all-clear, and a reader who wires it into CI as an acceptance test will read "clean" on most of what is not.
13 reST directives and 120 reST roles rendered as literal text (#664). mkdocstrings is configured docstring_style: google, under which .. warning:: renders as those exact characters and :func:`x` renders the literal :func:. Fifty-six of the roles reached the built site that way, across seven API pages. One of them was strip_format's markup-safety warning — a threat-model statement rendered as an ordinary paragraph with a stray directive at the front.

Converted to Warning: / Note: / Deprecated: sections and plain code spans, both of which render on the site and read correctly under help(). tests/test_docstring_conventions.py holds the convention, and checks its own patterns still match the forms they are for, so it cannot lapse into passing vacuously. RELEASING.md no longer tells contributors to write .. deprecated:: X.

The naming rule is written down (#654). A public name may describe the operation, never the outcome. canonicalize, not clean. It was being followed and re-argued each time someone proposed clean(); CONTRIBUTING.md now carries it, with the reason (NFKC unmasking makes output more dangerous to emit, so a name promising safety would be actively wrong) and the one shipped exception recorded — ml_normalize is named for a use case, which is why its docstring carries the warning above.

canonicalize also gains the sentence that closes the one real findability gap: "For cleaning untrusted input before comparison, this is the entry point. It does not make text safe to emit; encode at the sink." It contains clean, untrusted and safe, the three words a searcher uses, and promises none of them.

The Java and Kotlin bindings have documentation (#628). They shipped in 0.13.0 and the site never absorbed them: no docs/java/, no nav entry, and four lines in the whole tree mentioning the binding. A reader arriving from Maven Central had no supported path from the artifact to a working example.

Two pages, deliberately scoped to what no other page can give a JVM reader rather than mirroring all 69 methods. docs/java/getting-started.md covers the coordinates for Gradle and Maven, the JDK 21 floor, the five bundled natives, the exception hierarchy, and the two things with no counterpart elsewhere: Pipeline and Lexicon are AutoCloseable over native handles, and hasAnomalies has no single-argument form. docs/java/api.md covers the two call styles, the four options builders, the types, and a name mapping from the bindings a reader may already know.

Every example was run before it was written down, which is how three of them got fixed: hasAnomalies("...") does not compile, Kotlin's functions are top-level rather than members of a Disarm object, and the transliteration scheme is STRICT_ISO9 rather than ISO9.

BINDINGS.md carried Java as a planned binding with "JNI or Panama (FFM)" as an open choice, no Maven coordinates in the artifact table, and no Kotlin row at all. All three corrected, and the table's "as of 0.11" alignment note brought to 0.14. bindings/java/README.md now exists, so the GitHub directory view and the published POM url reach something.

Comparing the surface against generated/parity.yaml for that page turned up #677: the JVM has neither canonicalizeStrict nor stripFormat, so the two-call recommendation on the CVE page cannot be followed there as written — and the parity matrix does not track the JVM at all, which is why nothing had noticed.

Four doc sites told readers to import a name the package does not export (#660). LANG_AUTO is defined in disarm._enums and is the one LANG_* value of 84 that disarm/__init__.py never re-exports, so from disarm import LANG_AUTO raises ImportError on every released version. One of the pages framed it as the type-safe option, which is the opposite of what it is.

The pages now pass lang="auto", which works and which the same pages already showed alongside it, and each says why the constant is absent. Exporting a name is a new capability, and no other binding has an equivalent, so the export itself waits for a minor release — #660 stays open for it with the lockstep constraint recorded.

Two of the blocks carried , which is why nothing caught this. Both now run under Sybil, verified by planting an assertion failure and watching it fail.

With this the drift gate's allowlist is empty: every disarm name the documentation uses resolves on the published wheel, with no exceptions carried. The two original entries left the way the ratchet intends — one with the page that documented it (#655), one when the pages stopped making the claim.

[0.14.0] — 2026-08-28¶

Upgrade notes¶

Stored keys move. If you persist the output of search_key, catalog_key or sort_key, this release is a reindex event. Nothing about the API changed; the bundled tables and the presets built on them did. A key you wrote to disk last year will no longer compare equal to one you compute today, and no exception will tell you.

docs/RUST_API.md states the general principle — data-driven output is not semver-stable — but its list names normalize_confusables, strip_obfuscation, is_suspicious_hostname and the canonicalize* presets, and not the three key builders. Whether their output carries a stability guarantee is open (#644). Until it is answered, treat this section as the statement of record for what moved.

Measured against the published 0.13.0 wheel in a clean virtualenv, over 5,030 inputs — real words in 13 scripts plus random samples across U+0020–U+2FFF:

function	outputs changed	% of corpus
`sort_key`	497	9.9%
`canonicalize_strict`	491	9.8%
`search_key`	206	4.1%
`catalog_key`	206	4.1%
`strip_obfuscation`	137	2.7%
`canonicalize`	53	1.1%
`normalize_confusables`	49	1.0%
`transliterate`, `slugify`, `ml_normalize`, `fold_case`	0	—

The last row is the useful one: the four entry points most likely to be holding a stored value are byte-identical to 0.13.0 on this corpus. If you key on transliterate or slugify, you have nothing to do.

What moved, and why each is deliberate:

The key builders stopped leaking 134 characters that transliterate() deletes (#602). A Cyrillic soft sign was surviving into a supposedly-Latin key, and catalog_key then folded it onto Latin b. Russian words containing ъ or ь are the visible case: search_key("подъезд") was podъezd and is now podezd; catalog_key("Пьеса") was pbesa and is now pesa.
153 code points now reduce to an empty key where they previously reduced to a non-empty one — 59 of them letters, including the Cyrillic hard and soft signs. A character the table maps to nothing has no ASCII form; that is the table's decision, now applied consistently.
canonicalize_strict gained the eclipsing-mark rule (#615): a combining mark whose own script differs from its base's is now dropped, which is what closes CVE-2017-7833. That rule is why it moves as far as it does — 9.8%, second only to sort_key in the table above, and the largest move of any non-key entry point. The idempotency defect #638 fixed was introduced and fixed inside this cycle, so it never reached a release — if you are upgrading from 0.13.0 you were never exposed to it, and canonicalize_strict("C҉̧") was already stable there.
The confusable table gained rows. 31 are attacker-observed mappings TR39 does not carry (#597); the rest were being discarded at generation time by a filter that ran before the pass which would have made them valid (#593, #595). The visible one is the capital sharp S: normalize_confusables("STRAẞE") was STRAẞE on 0.13.0 — unfolded — and is now STRASSE, so STRAẞE and STRASSE finally collide, which is what a skeleton is for. Anything keyed on a word containing ẞ, Ꟗ or ꞵ moves.

Deciding whether it affects you — run this against your own corpus rather than trusting the percentages above, which are a property of the sample:

# in a venv holding the OLD version, dump keys for your real values
import disarm, json

json.dump({s: disarm.search_key(s) for s in my_values}, open("before.json", "w"))

# then upgrade and compare
before = json.load(open("before.json"))
moved = [s for s, k in before.items() if disarm.search_key(s) != k]

If moved is empty you can upgrade in place. Otherwise re-derive the stored keys before comparing new ones against them; there is no migration path that converts an old key into a new one, because the change is a re-romanisation and not a mapping.

Keys computed and compared within one process are unaffected either way.

has_anomalies flags a class it did not flag before. The compat_fold kind (#633) reports a token that mixes a Unicode compatibility form with ASCII — ａdmin, ｅxample.com, ＜script＞. If you alert on has_anomalies, expect new hits on input that was previously reported clean; canonicalize already folded all of it, so nothing you were cleaning changes, only what you are told about.

It is gated twice to keep ordinary text out — the token must carry an ASCII letter, and some non-ASCII character must fold to ASCII — so ＮＨＫ, Ｑ＆Ａ, １９９５年, kΩ, µF and 10㎏ are all silent. Measured at 0 false positives on a 16-sample corpus of Japanese typography and unit symbols. A token spelled wholly in a compatibility form (ｐａｙｐａｌ) is deliberately not flagged.

Node callers: the AnomalyKind union gained 'compat_fold'. An exhaustive switch over it needs a new arm.

Added¶

compat_fold — the last row on the CVE page a character class could close (#633). canonicalize("＜script＞") returned <script> and inspect_anomalies("＜script＞") reported clean: the whole class was neutralized and none of it was detected, the same asymmetry #603, #605, #610 and #612 each closed for a different character class.

It lands against an explicit prediction. #612's closing text argued the remaining rows each needed a comparison between two strings rather than the presence of a character, so no further character class would close them. Right about the others, wrong about this one: a compatibility fold is checkable per token — compare the token to its NFKC form — and what made it hard was never detection but false positives.

Two gates, and both were found by something firing rather than by design. The token must carry an ASCII letter, and some non-ASCII character must fold to ASCII. The first draft required only "changes under NFKC" and fired on kΩ µF resistor, which an existing test caught — the Ohm and micro signs fold to Greek and disguise nothing. The second draft required an ASCII alphanumeric and fired on 10㎏, which folds to 10kg; 125 code points in U+3000–U+33FF fold to ASCII, so the squared CJK units were a whole class. The disguise case is a word spelled half in a compatibility form, and a word has letters.

Reaches five CVE rows, and four of them are out of scope. CVE-2007-2688 (Cisco IPS fullwidth evasion) becomes Neutralized + detected; CVE-2019-9636, CVE-2024-43093, CVE-2023-41889 and CVE-2023-52081 keep their disposition — disarm does not parse URLs and cannot stop them — but a caller who screens before deciding is no longer told the input is clean. has_anomalies goes from 19 rows to 24, and the undetected-in-scope set from seven to six.

A token spelled wholly in a compatibility form (ｐａｙｐａｌ, １２３) is deliberately not flagged: by character class it cannot be told from ＮＨＫ, and a detector that fires on ＮＨＫ is one a CJK-facing caller switches off entirely.

find_key_collisions — which of these names are the same name (#620). The first disarm entry point that takes a collection. Every other detector is a single-string predicate, and a collision is not a property of a single string: groß.txt is an ordinary German filename, and аdmin is only a problem next to admin. This is the question node-tar's PathReservations guard failed to ask before extracting two paths into one slot in parallel (CVE-2026-23950), and the one a registry has to ask before accepting a second admin (CVE-2013-7236). Available in Rust (api::find_key_collisions, api::KeyForm, api::KeyCollision), Python, Node (findKeyCollisions), Ruby (Disarm.find_key_collisions), the C ABI and Java/Kotlin.

The reducer is the policy, and there is no default. Measured against the four collision rows in docs/security/cve-validation.md:

key	2026-23950	2019-19844	2013-7236	2020-12063
`fold_case`	yes
`search_key`	yes	yes	yes	yes
`catalog_key`	yes	yes	yes	yes
`canonicalize`		yes	yes	yes
`canonicalize_strict`		yes	yes	yes
`normalize_confusables`		yes	yes	yes

A stronger key finds more collisions, including ones nobody attacked: search_key collides Muller with Müller and Ivan with Иван. That is not a false positive — they really are one key — it is the cost of the key you chose, and choosing it for the caller would be choosing their threat model. sort_key is deliberately absent: a sort key exists to collide, so reporting its collisions would be noise.

Not dedup_batch(report=True), which the issue offered as the alternative. That helper dedups on the raw input as a performance optimisation and never collapses groß.txt into gross.txt's slot — the collision in the issue's example comes from transliterate, not from the dedup — so a report bolted onto it would carry one fixed reducer and miss three of the four rows above. It is also Python-only by a documented scope decision, and node-tar is a Node package.

The report cannot disagree with the collapse it describes, because both come from one pass over one reducer. A group is returned only when it holds two or more distinct inputs: the same string twice is the same name twice, which a reservation table already handles. Groups come back in first-appearance order rather than in hash order.

is_case_fold_stable (#619) remains the single-string half — it says a name may collide, never with what. This says with what.

is_case_fold_stable — ask whether a value is a stable identity key before you key a table on it (#619). Answers fold_case(x) == x.lower(). A False says some other string folds to the same value, which is the precondition node-tar's PathReservations guard missed in CVE-2026-23950: groß.txt and gross.txt are one path on a case-insensitive filesystem. Available in Rust (api::is_case_fold_stable, and on DisarmStr), Python (is_case_fold_stable, Text.is_case_fold_stable), Node (isCaseFoldStable), Ruby (Disarm.case_fold_stable?), the C ABI and Java.

It states a fact about the string, not suspicion. groß is an ordinary German word and ﬁle an ordinary ligature, so the predicate reads True for ordinary text and is deliberately kept out of has_anomalies and out of the CVE detector panel — folding it in would flag ordinary German and every Greek word ending in sigma. What to do about a False is the caller's decision: reserve both forms, reject the name, or key the table on fold_case rather than str.lower().

str.lower() is the comparison basis and str.casefold() is not. Casefolding performs the very transform under test, so a predicate written against it answers True for every string in Unicode — the substitution #617 already made once, now pinned by a test.

Not a per-character table, because a per-character table is wrong for Greek. ΟΔΟΣ ("street") lowercases to οδος and folds to οδοσ, yet Σ agrees with itself in isolation; U+03A3 is the only code point in Unicode whose lowercase mapping depends on its neighbours, which a Tier-3 test asserts by enumeration rather than by assertion. The implementation is allocation-free for anything that contains no capital sigma — ASCII short-circuits, everything else scans the folding table in place — and falls back to the exact string comparison for the rest, so it cannot drift from what fold_case actually does.

CVE-2026-23950's Detected by column stops reading —. The collision itself is still a property of a pair of names and no single-string predicate can report it (#620 tracks that); the precondition is what moved. Measured limit: the issue paired this with CVE-2019-19844, and that half does not hold — that row's probe turns on U+0131 DOTLESS I, which folds and lowercases to itself and collides through .upper() instead, so the predicate is silent on it.

Ten CVEs on encoding rather than code points, and the survey method behind them (36 → 46 rows). Found by sweeping NVD across the operations disarm performs, then verified one ID at a time.

Class	Added
Overlong / invalid byte sequences	CVE-2024-46954, CVE-2026-44288, CVE-2009-4142
Lone surrogates	CVE-2022-31116, CVE-2025-64439, CVE-2008-4066
Full-width evasion of a detector	CVE-2007-2688, CVE-2001-0669
Encoding layers disarm does not own	CVE-2022-3782, CVE-2006-2753

The byte-level rows are the first on the page whose input is not a str. decode_to_utf8 replaces overlong sequences rather than decoding them, so the Ghostscript traversal never materializes and strict=True refuses outright. CVE-2026-44288 names the correct behaviour exactly — protobufjs decoded overlong sequences "to canonical characters instead of replacing them" — which is what makes not-affected measurable rather than asserted.

CVE-2025-64439 is a Unicode edge case reaching RCE through an error path: illegal surrogates made msgpack serialization fail in LangGraph, and the fallback was JSON deserialization of untrusted data. disarm substitutes rather than drops, which is what keeps key<U+DC00>value from colliding with keyvalue — the CVE-2022-31116 shape.

CVE-2007-2688 is the Threat Model's ordering rule eighteen years early. Cisco IPS, Check Point and IBM ISS Proventia all shipped the same missing normalization step in the same month. It is also the same fold TestFullwidthUnmaskingHazard pins as a hazard — both readings are correct, and pipeline position decides which applies.

docs/security/cve-validation.md now records how rows are found: the NVD sweeps by mechanism rather than by product, the per-ID verification that caught CVE-2017-20190 having no CVSS at all, and the non-CVE research that informed rows — Paul Butler on variation-selector smuggling, and the CoreText Telugu crash.

The comparator table no longer contradicts the matrix. CVE-2026-23950 rendered as Neutralized in the matrix and as no under both disarm columns in the comparison a hundred lines below, because the comparison scores every row against two fixed disarm entry points and that row is neutralized by fold_case.

The columns stay fixed on purpose — decancer and unidecode each expose exactly one entry point, so letting disarm pick a different function per row would flatter it. Instead the three rows whose matrix neutralizer is neither fixed column now carry a † naming the function that does own them, so a no reads as not this function rather than not disarm. test_named_elsewhere_matches_the_registry derives that set from the registry rather than trusting the benchmark's copy, and also checks the named function is one the row actually lists.

Comparator corpus caught up with the matrix, and gated against falling behind again (15 → 22 of 36 rows). The corpus stopped growing when the matrix went from 20 to 36 rows, and the drift gate at the time only compared it against NEUTRALIZABLE — which had stopped growing too. Both sides moved together, so nothing failed and 21 rows fell out of the comparison silently.

Seven comparable rows are now compared: the four terminal-control CVEs (CVE-2025-55754, CVE-2024-52005, CVE-2023-43620, CVE-2023-37275), the zalgo row (CVE-2017-20190), the Latin kra (CVE-2019-11721) and the sharp-s path collision (CVE-2026-23950).

The other 14 are named rather than left unexplained: 11 out of scope (nothing neutralizes them, so nothing to compare), 2 not-affected (a cost property, not a transformation), 1 detected without being neutralized. test_every_registry_row_is_compared_or_has_a_reason_not_to now checks each row against the registry rather than against another list that can drift with it.

A flaw in the comparison predicate came out of adding the sharp-s row. The harness neutralized case with str.casefold(), which performs Unicode full case folding — and therefore maps ß to ss itself. Every tool would have passed CVE-2026-23950 by measuring Python rather than the tool. Switched to str.lower(), which leaves ß alone; no other row changed.

Scores on the widened corpus: canonicalize and strip_obfuscation 20/22, decancer 16/22, unidecode 14/22. The single row where disarm reads no and unidecode reads yes is CVE-2026-23950, and it is a deliberate refusal — folding ß in the confusable table would rewrite ordinary German, so the key builders own that collision. The page says so next to the table.

Normalization cost as a CVE class, and a fourth disposition to describe it (33 → 36). CVE-2026-3276 (CPython unicodedata.normalize() on alternating-CCC runs, CWE-407), CVE-2023-46695 (Django NFKC on Windows, re-reported three times since) and CVE-2017-20190 ("Zalgo text", disputed, deferred, and carrying no CVSS score at all — only SSVC).

The input here is not a disguise, it is a bill, and the existing vocabulary could not say what disarm's relationship to it is. not-affected now means the CVE is a defect in another implementation of something disarm also does, and disarm's implementation was measured and does not have it — distinct from out-of-scope, which means disarm does not stop the attack. It is a stronger claim, so it is gated: identical output to CPython across all four forms, and a linear-cost bound.

disarm is not uniformly faster, and the page says so. Over nine input shapes at 20,000 characters it runs between 6× faster (CJK compatibility ideographs) and 10× slower (already-normalized text, where CPython's quick-check short-circuits and disarm does more work). An earlier informal measurement suggested a ~700× margin; that was a cold-start artefact and does not survive best-of-N timing.

The bound is the actual defense. canonicalize collapses a 2,000-mark pile to at most four characters, and is_zalgo flags the CVE-2026-3276 payload too — rejecting the input costs less than normalizing it quickly. What disarm does not do is bound input length, which is what the Frigate/Yeti/spbu_se_site CVEs in this family are actually about; test_disarm_does_not_bound_input_length pins that distinction so the cap is not misread as a resource limit.

Eleven more CVEs, in the classes the matrix was thinnest on (22 → 33).

Class	Added
Ordering: normalize-then-validate	CVE-2026-28289, CVE-2024-43093, CVE-2023-41889, CVE-2023-52081
Terminal control sequences	CVE-2025-55754, CVE-2024-52005, CVE-2023-43620, CVE-2023-37275
Address bar and deny lists	CVE-2019-11721, CVE-2023-4399
Case-folding path collision	CVE-2026-23950

The ordering rows are out of scope on purpose. disarm can produce the canonical form; it cannot make a caller look at it first. CVE-2026-28289 states the bug in the Threat Model's own terms — NVD describes a TOCTOU weakness where "the dot-prefix check occurs before sanitization removes invisible characters". CVE-2024-43093 is in CISA's Known Exploited Vulnerabilities catalog.

The terminal class is neutralized and entirely undetected. All six escape-sequence rows are cleaned by strip_log_injection and reported by nothing, which is the sharpest argument yet for cleaning unconditionally rather than screening first. test_no_detector_reports_any_terminal_control_row pins that as a class-level claim.

CVE-2026-23950 closes a loop. node-tar's symlink poisoning turns on the ß/ss path collision — the single code point the CVE-2019-19844 exhaustive scan identified as the one the confusable table deliberately leaves alone, because ß is a real German letter. The key builders collide it; the canonicalizers correctly do not.

Two schema changes fell out of the additions. cvss and cvss_version are now optional, because CVE-2017-20190 has no CVSS record at all — only SSVC — and inventing a number would be worse than an empty cell. v4.0 joins the accepted versions.

One probe needed adjusting rather than one assertion: ASCII | is itself a TR39 confusable source, so a curl evil|sh payload tripped is_confusable for reasons unrelated to its CVE. The derived-detector gate caught it as a false positive.

Corrected: there is no single call that neutralizes every vector (#609 follow-up). The CVE page said canonicalize was the one call to make when the attack is unknown. That was wrong, and its gate could not catch it, because every vector in the matrix at the time happened to be one canonicalize handles.

CVE-2017-7833 (Firefox, 5.3) is the vector that breaks it: a single Arabic vowel mark riding a Latin letter. One mark sits below the zalgo threshold, so is_zalgo correctly returns False, and canonicalize caps combining marks rather than removing them (#429) — so the spoof never collapses onto the genuine host.

CVE-2017-5383 (Firefox, 5.3) is the mirror image, and rules out the obvious replacement. strip_obfuscation removes the mark, but renders punctuation confusables as their names — U+2010 HYPHEN becomes the word "hyphen" — so it never folds to ASCII -. Neither preset dominates the other and they fail on different inputs, so "5/6 each, pick either" is the wrong read.

Measured across the matrix plus both, no entry point clears everything. catalog_key comes closest — the only one carrying both a confusable step and strip_accents — and it has no format-stripping step, so the Tags block of CVE-2025-32711 goes straight through. The answer is a composition: canonicalize(strip_zalgo(text, max_marks=0)).

TestOneCall now gates both halves, including a test that fails if any single entry point ever does become sufficient, so the guidance is revisited rather than left stale. has_bidi_conflict also stops reading zero: CVE-2017-7833's Arabic mark beside Latin letters is exactly the strong-direction mix it asks about.

CVE matrix: comparator columns, split entry-point roles, and a measured "one call" answer (#607 follow-up). Three gaps in the published matrix, all of them the kind that make a table look more useful than it is.

canonicalize is the single call, and that is now measured rather than recommended. The matrix said which entry point handles which CVE; a defender's actual question is which call to make when the attack is unknown. canonicalize handles all thirteen neutralizable vectors, as do canonicalize_strict, strip_obfuscation and the llm_guardrail / rag_ingest profiles. strip_format handles eight — it has no confusable step, so every row needing a fold survives it. Every score is pinned in TestOneCallSuperset.

Detection has no such answer, and the asymmetry is the point. No detector covers the matrix, and neither does all of them together: CVE-2023-24329, CVE-2008-2383, CVE-2019-9535, CVE-2025-32711 and CVE-2019-9636 are silent to every one. All five are still neutralized by canonicalize, which settles the pipeline question — clean unconditionally, and use the detectors to decide whether to alert, never whether to clean. A pipeline that screens first and cleans only what it flagged forwards those five untouched.

Neutralizers and detectors are separate fields. They were one entry_points list, which read as though any name on it would defend the row. CVE-2019-19844 is the clearest case: its neutralizers detect nothing and its only detector rewrites nothing — and it was mislabelled Neutralized when it is also Detected. Each row's detector list is now derived, not written: the suite runs the row's vector through every detector and asserts the list matches what fired.

Comparator columns. benchmarks/cve_comparators.py runs disarm, decancer and unidecode over the same vectors under one predicate, and regenerates the published table. disarm 13/13, decancer 8/13, unidecode 10/13 — but the score is the least interesting part. unidecode maps by sound, so its homoglyph passes are decided by which homoglyph the attacker picked (рroduсt → rrodust, not product); decancer reorders bidi text rather than stripping it, leaving U+202E in the output. decancer-py==0.4.1 joins the bench extra.

CVE validation suite (tests/test_cve_vectors.py, docs/security/cve-validation.md). The docs encourage security use; nothing checked that against a named attack. Twenty published CVEs are now reconstructed from the vector each one describes and asserted against disarm's real behaviour, in the CI gate, across six classes:

Class	CVEs
Source code and identifiers	CVE-2021-42574, CVE-2021-42694
Identity and account takeover	CVE-2019-19844, CVE-2013-7236, CVE-2020-12063
Filesystem and paths	CVE-2014-9390, CVE-2009-3376, CVE-2023-33955
Hostnames and URLs	CVE-2017-7832, CVE-2023-24329, CVE-2019-9636
Terminal output and logs	CVE-2008-2383, CVE-2019-9535
ML / LLM input	CVE-2025-32711, CVE-2024-5184, CVE-2024-5565, CVE-2023-29374, CVE-2023-36258, CVE-2024-3098, CVE-2023-32786

Seven rows are out-of-scope negatives, and that is the point. A suite that recorded only wins would quietly convert THREAT_MODEL.md's "no guarantee that any class of attack is fully neutralized" into a coverage claim. The negatives are asserted so the limits cannot drift into untested marketing — including two rows where canonicalizing in the wrong pipeline position makes an attack worse: ＿＿ｉｍｐｏｒｔ＿＿ becomes executable ASCII under NFKC (CVE-2024-3098's safe_eval blocklist), and ＃ becomes a real # that moves where a host ends (CVE-2019-9636).

Four findings came out of measuring rather than assuming. has_bidi_conflict() is correctly False for every Trojan Source payload — they are ASCII plus controls, with no strong RTL run — so it is not the detector for that family. collapse_whitespace() leaves a leading NUL, so it does not close CVE-2023-24329 on its own. ᴀ (U+1D00) has no uppercase mapping at all, so it is not a CVE-2019-19844 vector despite looking like the obvious one; the real collision class (non-ASCII code points whose .upper() is pure ASCII) is exactly ten members wide, walked exhaustively over all of Unicode in ~0.2s, and nine fold at canonicalize_strict while ß closes only under fold_case. And there is no "old CVEs are CVSS v2" rule: NVD backfilled a v3.1 score for CVE-2014-9390 while leaving CVE-2013-7236 and CVE-2009-3376 v2.0-only, so each row records the revision it quotes.

ml_normalize() passes all twelve bidi controls, PUA, and homoglyphs through unchanged — it is a tokenizer-hygiene preset, not a screen. The llm_guardrail and rag_ingest profiles are the entry points for untrusted text, and that is now asserted rather than implied.

The suite is mutation-checked: neutering any of nine entry points turns it red, so no assertion passes vacuously. TestDocsMatrixDrift derives the published table's scores and disposition wording from the registry, so a row cannot be softened in Markdown alone.

31 real-attacker confusable mappings TR39 does not carry (#597). Miss-mining the BitCore subset of the BitAbuse corpus (Lee et al., NAACL Findings 2025) with benchmarks/adversarial_eval surfaced codepoints that attackers substitute for basic-Latin letters and that TR39 does not list as sources at all, so normalize_confusables left them unfolded. ɴ→n alone accounts for 4,576 real occurrences; the top three (ɴ, ʍ, ɾ) are 74% of the Tier-1 mass.

They arrive in a new file, data/confusables_attested.tsv, not in confusables_supplement.tsv. That file declares itself cross-script and pins its provenance to one measured dataset above a stated danger threshold (#336); 18 of these sources are Latin folding to Latin, and none comes from that dataset. Two admission criteria, two provenance stories, two files — the generator merges them, but the audit trails stay separate.

The contract widens, and that is deliberate. Tier 1 (23 rows) are optical twins. Tier 2a (7) are positional — an attacker reached for an Armenian, Georgian or runic glyph by its place in the word, not because it resembles the letter — and Tier 2b (1) is convention. Admitting 2a means this table now encodes observed attacker substitution, which is wider than visual confusability; Unicode would not accept those rows upstream. The tier is recorded per row, and the widened rule is stated in docs/user-guide/confusables.md and THREAT_MODEL.md rather than left implicit.

Two details worth knowing. µ U+00B5 is folded alongside μ U+03BC, because NFKC maps one to the other and folding only one would make the result depend on the input's normalization form. And six rows fold an uppercase source to a lowercase target (Ƿ Ʌ Ա Ⴝ Ⴍ Ⴓ), which no generated row does — the generated pipeline reconciles case to the source and these bypass it. The attested form is kept, because the evidence is the letter the attacker meant, not its case.

Per the #39/#40 guardrail the corpora are measuring instruments, never optimization targets: the synthetic BitViper tail — 254 further codepoints, 733,029 occurrences, 98.5% of all novel misses — is excluded by construction, and no row is justified by a benchmark score. Latin table 2,189 → 2,220 mappings; the Cyrillic table is untouched, since seven of these sources already fold there and the new rows set the Latin column only.

The C header is committed and drift-gated (#580). bindings/cabi/disarm.h was gitignored and regenerated inside the CI step that then compiled smoke.c against it. Both sides therefore moved together: a signature change plus a matching call-site change was self-consistent and passed. That is how a widened disarm_normalize_confusables (2 args → 3) reached review on #574 with every check green — a human reading the diff caught it, no test did.

The header is now a committed baseline, the same way src/metadata.rs and generated/parity.yaml are committed and drift-checked rather than regenerated blind. CI diffs the regenerated header against it in the cabi job, so every ABI change is a visible diff someone has to approve. A second test pins the arity of the entry points that shipped before 0.14, so a widening fails by name rather than as one line in a 346-line diff.

The gate does not judge whether a change is breaking — it makes changes visible. Deciding that a diff is additive (a new _opts entry point) rather than breaking (a widened signature) stays the reviewer's job.

Selectable digit-mapping policy (#561). disarm folds a non-Latin digit to the ASCII digit; upstream TR39 folds several of them to a Latin letter (० → o, ೦ → O, ١ → l). Neither is wrong — numeric is right for prose, where a Devanagari zero really is a zero, and the letter is right for an identifier skeleton, whose only job is to make two confusable identifiers collide. The divergence was fixed in the table with no way to select the other side, so it read as a defect to anyone scoring disarm against a TR39-derived benchmark and cost points silently.

digit_policy now selects it: "numeric" (default, unchanged behaviour) or "tr39". Reaches Python (normalize_confusables(..., digit_policy=…) and Text), Rust (api::normalize_confusables_with + the DigitPolicy enum), Node (digitPolicy option), Ruby (digit_policy: keyword), Java/Kotlin (DigitPolicy enum), and the C ABI.

The Rust surface adds a second function rather than a third parameter on normalize_confusables: that is the crate's most-used security primitive and the policy is rarely set, so widening it would tax every call site for something almost none of them need. normalize_confusables(text, target) is unchanged.

The divergent rows are generated, not hand-maintained: scripts/gen_confusables.py already computes both sides — it makes this exact choice at generation time via enforce_digit_target (#439) — so the discarded alternative is now emitted as src/tables/data/confusables_digit_tr39.tsv (45 rows) and build.rs turns it into an override PHF. An override set rather than a second full table, so the two policies cannot drift on the rows they agree on, which is all but 45 of them.

Scope: the policy is a property of the normalize_confusables entry point only. The presets (canonicalize, catalog_key, search_key, …) serve prose and keys, where numeric is unambiguously right, so they have no switch. Hostname analysis also stays numeric — selecting TR39 there would silently change what is_suspicious_hostname flags, which is a security-behaviour change and belongs in its own issue.

Scope, second axis: "tr39" applies to the Latin target only; with any other target script it is a no-op and the fold behaves exactly as "numeric". The override set is generated from the Latin table and its values are TR39's Latin-script targets, so consulting it under target_script = "cyrillic" emitted Latin letters into a Cyrillic skeleton (० folded to o, not 0) and invented folds for sources the Cyrillic table deliberately has no row for. Three of those leaked outputs (Ʌ, o, rn) are themselves confusable with Cyrillic, so the fold did not even reach a fixed point.

ml_normalize reaches every binding. It was Rust + Python only, recorded in scripts/parity.py as a deliberate scope decision. That decision does not survive contact with what the preset is for: it is the ML/NLP entry point, so keeping it Python-only meant a Node or JVM model pipeline could not use disarm for the thing disarm built it to do. Now exposed as mlNormalize (Node, Java/Kotlin), Disarm.ml_normalize (Ruby), String.mlNormalize (Kotlin extension), and disarm_ml_normalize (C ABI), each with the fold_case switch from #559.
Multi-codepoint confusable sources — contraction (#562). The confusable tables map one codepoint to one-or-more (0271 → rn), so expansion always worked. Contraction — recognising that rn may stand in for m — could not be expressed at all: the source column of both TSVs is a single hex codepoint in every data row. This was a schema change before it was a data change.

is_suspicious_hostname(host, contractions=True) / api::analyze_hostname_with(host, true) now folds ASCII digraphs that can impersonate a single letter into the canonical form, so arnazon.com canonicalizes to amazon.com. Reaches Python, Rust, Node, Ruby, Java/Kotlin, and the C ABI.

Off by default and confined to the hostname path. Unconditional contraction is worse than none: rn → m is right for arnazon and wrong for earnings, turnip, born. A hostname is the one place where the threat model justifies those false positives and there is no running prose to corrupt. It is not reachable from normalize_confusables at any setting; a general-text mode would need its own disambiguation story.

Three rules, each with recorded provenance: rn → m is the one TR39 itself sanctions (it reduces m to the sequence rn, and 17 sources fold to rn, the dominant multi-character target in the file); vv → w and cl → d are disarm additions from the IDN homograph literature. Every rule is a false-positive source, so the bar is "documented real-world technique", not "plausible".

Matching is leftmost-longest over an Aho-Corasick automaton (reusing the dependency #242 already brought in), and applied per label, so a digraph can never form across a dot. One pass is a fixed point by construction: build.rs asserts no rule's output occurs inside any rule's input, so a pass cannot expose a fresh match, and a data edit that introduced such a chain fails the build.

Argument style follows each ecosystem: an options object in Node/TypeScript, keyword arguments in Ruby, a MlNormalizeOptions builder in Java (mirroring the existing TransliterateOptions), default parameters on the Kotlin extension, and positional arguments with a nullable lang in the C ABI.

ml_normalize is removed from SCOPE_REVIEW in scripts/parity.py; the op now reads ✓ across rust/python/ruby/node in the parity matrix.

Nightly Hypothesis run (.github/workflows/nightly-hypothesis.yml). Tier 2 is excluded from PR CI on purpose (~440 tests, non-deterministic, slow) and is not in the Tier-3 release gate either, so it ran only when a developer happened to run the full suite locally. #570 sat undetected in exactly that gap.

Runs at 03:17 UTC and on demand, with --hypothesis-seed=random so consecutive nights explore different input space, and ORACLE_MAXEX=20000 — 10× the local default — for the adversarial-oracle suite, the one env-tunable budget in the tier and the suite that found #570. A failure opens (or comments on) a single rolling issue rather than disappearing into a run log.

Deliberately not a required check and not in the publish path: a probabilistic suite must never be able to block a security release. It reports; a human triages.

The bundled confusables.txt version is readable at runtime (#560). Nothing in the library reported which upstream release the confusable tables were folded from. The number existed — in the TSV header and in docs/provenance.md — but both are build-time artifacts, so a deployment could not answer "is my fold stale?" without inferring it from behaviour. It is now exposed everywhere: disarm::api::CONFUSABLES_VERSION (and api::confusables_version()) in Rust, disarm.CONFUSABLES_VERSION in Python, confusablesVersion() in Node and Java/Kotlin, Disarm.confusables_version in Ruby, and disarm_confusables_version() in the C ABI.

build.rs parses the value out of the TSV header it already reads, so the constant cannot drift from the data it describes, and the build fails if that header stops naming a version. Both confusable tables are folded from one upstream release, which build.rs asserts, so a single constant covers them.

There is deliberately no library-wide UNICODE_VERSION: disarm's bundled tables track different releases (confusables 17.0.0, case folding 16.0, East Asian width 15.1.0), so one number would be wrong for three of the four. See docs/provenance.md for the full table and the per-language accessors.

ml_normalize takes fold_case=False (#559). The preset folds case deliberately, and that is defensible — most tokenizers are uncased. What was missing was a way to turn it off for one call. A caller who wants everything else the preset does (NFKC, demojize, transliterate, strip-accents, control and zero-width removal, whitespace folding) in front of a cased model now has a route to it. The fold is destructive and cannot be undone downstream, and an uncased evaluation harness cannot measure what it costs.

Default true/True, so existing behaviour is unchanged. The flag drops Step::FoldCase and nothing else; the no-fold step list is derived from the folded one by a const fn, so the two cannot drift, and a build-time assertion fires if the pipeline ever stops containing exactly one fold step.

ml_normalize is Rust + Python only (Node/Ruby/Java/C-ABI do not expose it — a standing scope decision recorded in scripts/parity.py), so the flag reaches disarm::api::ml_normalize, disarm.ml_normalize, and Text.ml_normalize.

Note fold_case=False restores case, not diacritics: strip_accents is a separate step and still runs, so José becomes Jose. See below.

Confusables coverage introspection (#563). find_untranslatable has existed for transliteration since #184; there was no confusables analogue, so answering "which sources does disarm not fold?" meant building a harness outside the library against a cached copy of confusables.txt. Two read-only accessors now answer it from inside:
unmapped_confusables(target) — every source in the bundled upstream file that the chosen table does not fold, sorted.
find_unmapped_confusables(text, target) — the same question for one input, in the shape of find_untranslatable: (char, byte_offset) in order of appearance.

Both reach Rust, Python, Node, Ruby, Java/Kotlin, and the C ABI.

The denominator is generated: scripts/gen_confusables.py now emits src/tables/data/confusables_upstream_sources.tsv (the source set it already read and discarded), and build.rs turns it into a PHF set. The exposure set is derived at runtime — upstream sources minus the resolved table's keys — so it cannot go stale against the table it describes, and one denominator covers both targets. The existing confusable TSVs regenerate byte-identically; this change is purely additive.

The per-input scan composes exactly as the fold does (#475/#477/#483), so a decomposed homoglyph whose precomposed form is mapped counts as covered, and offsets anchor to the caller's string rather than to the composed intermediate — matching find_untranslatable's guarantee.

Read the result as exposure, not as a score: a tool at 95% per-source coverage is one query away from the other 5%, and this set is where an adaptive attacker goes. Nothing is filtered out, which means the Latin set contains five ASCII characters (%, 0, 1, I, m) — TR39 is a skeleton transform (m→rn, I/1→l, 0→O) and disarm deliberately does not apply those rows, so a scan over ordinary English reports the letter m. Documented on every surface; a coverage report that quietly drops rows reads as coverage it does not have.

Changed (breaking)¶

disarm::api::ml_normalize gains a fourth parameter, fold_case: bool. Rust callers must add true to keep the current behaviour: ml_normalize(text, lang, emoji_style) → ml_normalize(text, lang, emoji_style, true). Python, which takes it as a keyword with a default, is unaffected. (#559)

Fixed¶

canonicalize_strict was not idempotent (#638). f(f(x)) != f(x) for a class of inputs #615 created. canonicalize_strict("C҉̧") returned Ç, and applying it again returned C — a comparison key that depends on how many times you applied it, which is the one thing a comparison preset must not have.

U+0489 has ccc 0, which makes it a starter: it blocks C + U+0327 from composing, so the confusable fold's fixed point correctly finds nothing to do. The #615 cross-script mark strip then removes it — a Cyrillic mark on a Latin base is exactly its target — leaving the two adjacent, and the terminal NFC composes them into Ç, which folds to C. One pass too late.

So the two steps expose work for each other in both directions. #615 reasoned about one: the fold rewrites the base, so a mark that matched beforehand can stop matching afterwards, which is why the strip goes second. This is the other: the strip removes marks, which can expose a composition the fold has already finished with. Neither ordering is a fixed point alone, so they now iterate together. It converges because every pass either folds a character or deletes a mark, and neither is undone.

Measured: canonicalize_strict only — canonicalize and strip_obfuscation have no cross-script mark step. 474 code points reach the shape in the C + X + cedilla probe alone; they are the composition-blocking starters that are also script-specific marks (U+0488, U+0489, the Thaana vowel signs, and others). Verified over ~6.8M probes of base × code point × mark, with zero non-idempotent results.

Found by canonicalize_strict_idempotent on CI, at 569 successes — the same proptest that caught #615's first ordering attempt. The failing seed is now committed so it fails deterministically rather than randomly.

Implemented as a dedicated pipeline step rather than the generic FixedPoint combinator, which allocates per inner step per pass and took canonicalize_strict from 6 allocations per call to 12 — preset_alloc_count refused it. The dedicated step reuses buffers and exits after the first strip when the strip changed nothing, so text with no cross-script mark (essentially all text) pays nothing for the loop.

The two CVE rows behind "no single call" are closed, and they had to close together (#614, #615). Each was one of the exactly two vectors that made docs/security/cve-validation.md say no entry point cleared everything, so fixing one alone would have left the other failing and forced the guidance to be rewritten twice.

#614 — strip_obfuscation named confusables instead of folding them. 49 code points appear in both emoji_single.tsv and confusables_to_latin.tsv, and most are not emoji: typographic punctuation, currency, math operators, CJK brackets. They reach the emoji table from CLDR annotationsDerived, which names non-emoji characters. strip_obfuscation("€xample.com") produced "euro xample.com", so the spoof and the genuine host stopped being equal rather than becoming equal — CVE-2017-5383 surviving a preset documented as maximum-strength deobfuscation.

Not fixed the way the issue proposed. Reordering the confusable fold before demojize would break idempotency: punctuation inside emoji names (the ’ in "woman’s hat") has to be folded by the confusable pass. That ordering is documented three times and pinned by tests/test_presets.py. Instead the overlap is derived at build time as an intersection of the two tables — so it cannot drift the way a curated list would — and demojize skips those rows inside comparison presets only. Standalone demojize("I ❤ €5") still returns "I red heart euro 5", which is what that function is for. build.rs asserts the count is 49, so a table refresh that claims another confusable source fails the build instead of widening the gap silently.

#615 — canonicalize cannot cap its way out of an eclipsing mark. The anti-zalgo step is a count, and by count one Arabic shadda is indistinguishable from one acute accent, so no threshold removes CVE-2017-7833's spoof and keeps café. The discriminator that works was already in disarm's script data: strip a combining mark whose own Script is a specific script differing from its base's, and keep Inherited marks, which attach to anything. That is UTS #39's mixed-script reasoning applied per grapheme rather than per string.

It runs in canonicalize_strict only. The rule is destructive for scholarly transliteration, IPA and linguistic transcription, where marks from one script legitimately sit on bases of another — the corpus least able to notice. canonicalize is deliberately still one short, and there is a test asserting that rather than leaving it implied. Verified against nine legitimate samples in five scripts, including Arabic with its own vowel marks: all pass through completely unchanged.

The step sits after the confusable fold, and that ordering is load-bearing. Placed before it, а (Cyrillic) + U+0489 (Cyrillic mark) agrees on the first pass, then the fold rewrites the base to Latin a and the next pass strips the mark — f(f(x)) != f(x). The property test canonicalize_strict_idempotent caught it; deciding against the final base script is the only stable point.

canonicalize_strict and strip_obfuscation now each clear the whole matrix, so TestOneCall's guard is inverted rather than deleted: it asserted that closing a gap should fail loudly, it did, and it now asserts the two sufficient entry points stay sufficient while every other one stays short. The published advice is unchanged and its reason has moved — from "nothing suffices" to "the two that suffice are the two most destructive ones", which is the same conclusion for a caller who has to forward the text they cleaned.

is_suspicious_hostname() now catches tags, variation selectors, noncharacters and PUA, and stops reporting a noncharacter as Arabic (#610). Third in the sequence after #603 (bidi controls) and #605 (zero-width). 17 of 18 sampled code points passed the screen clean and all 18 survived into canonical.

The one that did flag was flagging for the wrong reason, and it is the #605 bug in a class #605 did not cover: U+FDD0 sits in the Arabic Presentation Forms range, so the script detector read it as a letter and paypal<U+FDD0>.evil.com reported scripts=['Latin', 'Arabic'] with mixed_script=True. Widening the existing per-label strip fixes that by construction rather than by special case, because the strip already runs before detect_scripts.

Reported on the existing has_invisible field rather than four new ones, so no binding payload changes: no new field on the Ruby positional tuple and no change to the Java jni_sig! string. is_invisible_in_hostname composes the four class predicates that src/invisibles.rs already carried.

Private use and the variation selectors are included because RFC 5892 puts all four of the classes added here in DISALLOWED outright. (The pre-existing zero-width set is not uniformly disallowed — U+200C/U+200D are CONTEXTJ, conditionally permitted — and flagging those remains the deliberate fail-closed policy #605 chose, not a reading of the RFC.) Both have legitimate uses in ordinary text, so a general-text detector needs its own argument for them; that is tracked separately. Measured against the full suite including the adversarial-oracle clean corpus: no false positives.

The tag block is the reason this is a security fix rather than tidying. U+E0061–U+E007A spell arbitrary Latin invisibly, and the screen previously called such a hostname clean and returned the payload intact in canonical — the combination that turns a detector into a laundering step.

search_key, catalog_key and sort_key no longer keep 134 characters that transliterate() deletes (#602). A character the table maps to the empty string is not unknown — it is a decision the table already made, "this has no ASCII form, drop it". ErrorMode::Preserve was excepting itself from those mappings on the reading that an empty mapping is a kind of failure the caller asked to keep, so the three presets that pass Preserve kept the characters verbatim while TextPipeline(transliterate=True), which passes Ignore, dropped them correctly.

catalog_key made it worse by running the confusable fold after transliteration, so a leaked Cyrillic soft sign was folded onto Latin b: Пьеса became pbesa, a key containing a letter that appears in neither the input nor its romanisation. It is now pesa.

ErrorMode still governs what happens to characters the table has nothing to say about, so a genuinely unmapped code point is preserved exactly as before. Verified by a full-range scan reproducing the issue's own predicate: zero leaks remain.

One property test asserted that Preserve never returns empty output. That held only because of the exception removed here, and could not have been true in general — a string of nothing but empty-mapped characters legitimately transliterates to nothing, which is why its generator already had to exclude combining marks. It is restated as the invariant that actually defines the mode: Preserve output is never shorter than Ignore output, which needs no generator exclusions at all.

A new control anomaly kind — has_anomalies goes from 11 CVE rows to 18 (#612). A non-whitespace control (NUL, ESC, BEL, DEL, the C1 block) is never legitimate in text, and nothing reported one. strip_control_chars has removed them since #433, so the transform existed and the detector did not.

The reason they were invisible is worth recording: the introducers are plain ASCII, so the ASCII fast path in the token classifier — which exists because the invisible, bidi, zalgo and mixed-script branches can only fire above U+007F — skipped them entirely. The new branch runs before that gate.

Presence, not position. #612 framed this as an "edge" question because it started from whitespace trimming, but a control hides things wherever it sits: the last character of "malicious\u001b\\" is a backslash, so an edge-only rule would call that token clean while the escape introducer sits one place in.

The whitespace-class controls are excluded, reusing is_fold_whitespace rather than restating the set. TAB, LF, VT, FF, CR, U+001C–U+001F and NEL are real separators that collapse_whitespace folds to a space, and flagging them would fire on every multi-line string.

This closes seven rows that docs/security/cve-validation.md listed as reported by nothing: CVE-2023-24329 (leading NUL) and the whole terminal-control class (CVE-2008-2383, CVE-2019-9535, CVE-2025-55754, CVE-2024-52005, CVE-2023-43620, CVE-2023-37275). The three that remain undetected are a different shape — a fold collision, a length budget, a table lookup — so no further character class will close them, and the page now says so.

Deliberately not added: leading/trailing whitespace detection, which #612 also asked for. inspect_anomalies documents itself as flagging characters "disguising a real word", and padding disguises nothing; a kind for it would fire on ordinary text.

The Node AnomalyKind union shipped without bidi_mixed. It was added to the Rust enum in #412 and never mirrored, so a TypeScript caller matching on it got a type error for a kind the library really returns. Nothing caught it, because the value crosses napi as a bare String and index.ts casts. Node is the only binding that restates the set — every other surface passes it through as a string — so a drift gate now reads the as_str arms out of src/anomalies.rs and compares them to the union, plus a second test asserting every kind is reachable from some input.
PRESETS["ml_normalize"] was missing two of the nine steps it claims to describe (#600). PRESETS is a hand-maintained Python mirror of the const STEPS arrays in src/presets.rs; nothing executes it, and it had drifted. The mirror listed seven steps, omitting the transliterate step and the second demojize that #498 added after strip_accents. test_preset_steps_exact did not catch it because it compares the mirror against a literal in the test file, and both were written from the same wrong reading — so the test pinned the drift instead of detecting it. Mirror and test are now correct against Rust.

list_profiles() also said presets are "step-lists defined in Python". They are defined in Rust. That sentence is how the drift went unnoticed, so it is corrected too, and a comment on PRESETS now states what the dict is and what it is not.

A structural gate that parses the Rust arrays is not part of this change: the step lists use composite variants (FixedPoint, ConfusablesNfcFixedPoint) that the mirror flattens, so a real gate needs per-preset expansion rules and deserves its own issue rather than a fragile parser bolted on here.

Documentation: three functions whose names promise more than they check.
has_bidi_conflict now says plainly that it is not the RLO check (#599). It reads letters, so "invoice\u202Egpj.exe" returns False — the two conditions are disjoint and a string can satisfy either, both or neither. The docstrings route to inspect_anomalies (kind bidi) for detection and strip_bidi for removal, and note that strip_bidi does not close the real-letter case, because there is no format character to remove. docs/concepts/which-function.md gains the two bidi rows its threat-model table lacked; its only previous mention of bidi was in a cost column, so the page could not answer "how do I detect a bidi attack".
get_pipeline() now states that profile names and PRESETS keys are disjoint namespaces, so get_pipeline("canonicalize") raising is expected rather than a bug (#600).
ml_normalize's documented limits stopped at homoglyphs. They now cover the other two: all twelve bidi controls and every PUA code point pass through unchanged (#608). strip_control handles Cc; bidi controls are Cf. No behaviour change — the preset is tokenizer hygiene, and llm_guardrail / rag_ingest already exist for untrusted input.
Python can now call strip_control_chars and strip_zero_width_chars directly (#616). They already existed in the Rust core (disarm::api) and in the C ABI, Java/Kotlin, Node and Ruby bindings. Python was the only surface without them, so control-stripping there meant constructing a TextPipeline rather than calling a function — unlike the ten sibling strip_* operations, four of which (strip_tags, strip_pua, strip_noncharacters, strip_variation_selectors) are narrower and are plain functions. Both are now exported, and Text gains the matching fluent methods.

The parity matrix recorded the gap as deliberate and named the substitute as collapse_whitespace(strip_control=True) — a signature that has never existed; collapse_whitespace takes only text. That record lived in PROVIDED_VIA in scripts/parity.py, so anyone consulting the matrix for the Python equivalent was sent to a TypeError. Both entries are removed and the matrix regenerated.

collapse_whitespace gains a property test covering control characters. The existing no_leading_trailing_whitespace property draws from \PC*, which excludes controls, so the trim invariant was never tested against them. It holds: measured exhaustively over the cross product of whitespace, controls and letters for lengths 1–4, and over 200,000 random strings, with zero cases where the output starts or ends with whitespace. Reported as a trim bug in #612; that report was wrong and is retracted there. What looked like a defeated trim is the space between a leading control and the word, which is interior by the same rule that makes "a\u{0}b" keep both of its spaces. No behaviour change — the test closes the coverage gap that made the question open.
is_suspicious_hostname() now catches zero-width and invisible characters, and no longer reports a phantom script for them (#605). Sibling of #603, for the characters that carry no direction at all — U+200B–U+200D, U+2060–U+2064, U+FEFF and U+180E. Eight of the ten passed the screen clean, and all ten survived into canonical.

The two that did flag were flagging for the wrong reason. U+FEFF sits in the Arabic Presentation Forms block and U+180E in the Mongolian block, so the script detector read each as a letter: paypal<BOM>.evil.com reported scripts=['Latin', 'Arabic'] and mixed_script=True. Right verdict, wrong evidence — and any caller keying policy on scripts was told an ASCII-looking hostname contained Arabic.

A new HostnameAnalysis.has_invisible reports the finding and is folded into suspicious. It is additive and disjoint from bidi_control (#603). The characters are removed per label, before script analysis, not on the joined hostname afterwards — so scripts, mixed_script, has_confusables and canonical are all computed on what a reader actually sees, and the phantom-script bug is fixed by construction rather than special-cased.

U+200C ZWNJ and U+200D ZWJ are flagged unconditionally. IDNA2008 CONTEXTJ permits them only in narrow joining contexts that a spoof screen has no reason to honour.

Exposed across all six surfaces (Rust core, Python, Node, Ruby, Java/Kotlin, C ABI).

is_suspicious_hostname() now catches bidi control characters, and canonical no longer carries them (#603). Every UAX #9 bidi control — the overrides U+202D/U+202E, the embeddings U+202A–U+202C, the isolates U+2066–U+2069 and the marks U+200E/U+200F/U+061C — passed the hostname screen clean, paypal<RLO>moc.evil.com among them. The verdict was derived entirely from bidi_conflict (#412), which reads strong-direction letters and is structurally blind to a format character.

The ACE path was never affected: idna::domain_to_unicode rejects these codepoints and the decode failure already failed closed. What slipped through was the literal-Unicode label, which reached the pass-through arm uninspected — exactly the form a hostname takes in a log line, a mail header or a UI label, where the name never resolves and the display spoof is the whole attack.

Two changes. A new HostnameAnalysis.bidi_control field reports the finding and is folded into suspicious; it is additive and disjoint from bidi_conflict, whose #412 meaning is unchanged (a string can set either, both or neither). And the controls are stripped before canonical is built, so a caller who screens a hostname and then renders that field can no longer render the spoof they were told was absent.

Exposed across all six surfaces (Rust core, Python, Node, Ruby, Java/Kotlin, C ABI). The character set is now defined once, in scripts::is_bidi_control; presets::is_bidi_or_format was refactored to build on it rather than keep a second copy, so the hostname screen and strip_bidi cannot drift apart.

ẞ uppercases to SS, not B — German is no longer corrupted (#597). #595 recovered ẞ (U+1E9E, the capital sharp S — official German orthography since 2017) by folding its TR39 prototype ß through ASCII_FOLD to b, then reconciling case. That produced B, so STRAẞE became STRABE, GROẞ became GROB and FUẞBALL became FUBBALL.

ß.to_uppercase() is the two-character SS, and that is the right answer: STRAẞE and STRASSE are the same word, so folding to SS makes them collide — which is what a skeleton is for. A genuine multi-character case mapping now wins over ASCII_FOLD. Two rows move, ẞ and Ꟗ (Middle Scots S, same prototype). Ᏸ U+13F0 is the counter-case that keeps the rule honest: a Cherokee letter shaped like B with no case expansion of its own, so it still folds to B.

tests/test_accented_latin_fidelity.py missed this because its German fixture is lowercase Straße, which was preserved throughout. It now carries the uppercase forms, the lowercase asymmetry, and the Cherokee counter-case.

Six Latin homoglyphs now fold, and the Latin lambdas collide with the Greek one (#593). filter_latin_homoglyphs recovers a Latin-script source whose TR39 prototype is a single basic-ASCII graphic. ASCII_FOLD runs later, in generate_mappings, so a row whose prototype that table already knows was discarded before it could be consulted — the same shape as #587 and #590: a filter dropping a row before the pass that would have made it valid.

source	prototype	now
`ẞ` U+1E9E	`ß`	`B`
`Ꟗ` U+A7D6	`ß`	`B`
`ꞵ` U+A7B5	`ß`	`b`
`ꝫ` U+A76B	`ȝ`	`z`
`Ꟛ` U+A7DA	`Ʌ`	`A`
`Ƛ` U+A7DC	`Ʌ`	`A`

TR39 puts ٨ ۸ Λ Ꟛ in one confusable class, and the Latin lambdas did not collide with the Greek one they are defined to be confusable with — the class had three distinct skeletons. It reads A throughout now.

Two candidates are deliberately excluded. ţ (U+0163) and ț (U+021B) reach the same prototype, but folding them strips a cedilla and a comma-below; ț is ordinary Romanian orthography, and normalize_confusables promises accented Latin comes through intact. The guard tests the source's own canonical decomposition rather than a codepoint list, so a future confusables.txt cannot smuggle a new accented source past it. It applies only to the rows this pass newly recovers: Ç, ç and Ǿ reach a bare ASCII prototype only because strip_combining removed the mark from TR39's target, and they have folded since long before this — #586's fixed-point loop is built on Ç → C.

Cherokee YE folds to B, not SS (#593). fix_case_mismatch uppercased the prototype before ASCII_FOLD could see it, and ß.upper() is the two-character SS, which then escaped the fold because that only fires on a single character. In a table about visual confusability, Ᏸ (U+13F0) is a B-shape. Both call sites now fold to ASCII before reconciling case; the blast radius was measured at this one row.

Latin table 2,183 → 2,189 mappings. The count gate added in #591 caught all five documented figures immediately, which is what it was for.

The Kotlin extensions keep their published JVM signatures (#588). A Kotlin default argument compiles to one JVM method plus a synthetic $default bridge, not to an overload per arity. Adding a defaulted parameter therefore deletes the signature that shipped, and anything compiled against the previous artifact gets NoSuchMethodError — Java callers, and Kotlin callers that have not been recompiled.

It had happened twice, unnoticed both times. dev.disarm:disarm-kotlin:0.13.0 published normalizeConfusables(String, TargetScript) and analyzeHostname(String); #574 and #562 each added a default and removed one. #562 made the identical break in the C ABI, where it was caught and reverted by #580 — the C surface has a committed, drift-gated header and this one had nothing.

All seventeen public extensions with default arguments now carry @JvmOverloads, which restores both lost signatures and preserves every arity from here on. JvmSignatureTest is the gate: it reads the compiled facade by reflection rather than the source text, so adding a parameter without the annotation turns it red. The policy is recorded in BINDINGS.md.

The cost is generated methods rather than maintained ones — slugify has twelve defaults and emits thirteen arities, taking the facade to 105 public static methods.

The tr39 digit-policy overrides now honour the ASCII contract (#587). #341 made ASCII the contract for the Latin confusable tables. The override set was written later, for #561, and never joined it: write_digit_tr39_overrides took upstream's raw target with only strip_combining applied, bypassing the ASCII_FOLD pass every value in the main table goes through. So digit_policy="tr39" put back exactly the residue #341 had removed.

Four of the 46 rows carried a non-ASCII value. Three had a clear ASCII representative and now use it:

source	TR39 target	now
`٨` U+0668	`Ʌ` U+0245	`a`
`۸` U+06F8	`Ʌ` U+0245	`a`
`⁰` U+2070	`º` U+00BA	`o`

This was not only cosmetic. TR39 puts ٨ ۸ Λ Ꟛ in one confusable class, and the un-folded value made the class stop colliding: ٨ gave Ʌ, Λ gave A, Ꟛ gave itself. Three skeletons for one class defeats the only thing a skeleton is for. After the fix the class reads a, a, A, Ꟛ — the digits now collide with the lambda case-insensitively. Ꟛ (U+A7DA) is still unmapped, for an unrelated reason: filter_latin_homoglyphs only recovers a Latin-script source whose prototype is basic ASCII, and Ʌ is not, so that row is dropped before ASCII_FOLD is ever consulted. That is its own gap, not this one.

The fourth, ⁹ → ꝰ (U+A770 MODIFIER LETTER US), has no clear ASCII representative. Rather than ship the residue, the row is dropped and tr39 falls back to the numeric reading for that codepoint, so ⁹ folds to 9 under both policies. The override set is 45 rows, every value ASCII, and build.rs now asserts it — the assertion the sibling table blocks already carried and this one did not.

The documentation claim is corrected too. Every surface said tr39 "folds several digits to a Latin letter"; three of the 45 rows do not land on a letter — ٠ and ۰ fold to ., and 𑣣 folds to the two characters rn. That is now stated wherever the policy is documented, because a caller building a label- or path-shaped key needs to know a delimiter can appear.

normalize_confusables now reaches a fixed point in every binding, not just Python (#586). 0.11.1 shipped #523 as "normalize_confusables is now idempotent and complete on confusable + combining-mark input". That was true of one of the two call paths. #361 had already wired the public Rust API to the single-pass normalize_confusables_cow a month earlier; #523 added the fixed-point loop to the owned Layer-1 form and its tests, and never touched src/api/safety.rs. Python reaches the core through the fixed path. Rust, Node, Ruby, Java, Kotlin and the C ABI reach it through the other one.

So the same call answered differently depending on the language, and the non-Python answer could still be confusable:

Input	Before, outside Python	Python, and now everywhere
`U+04AA U+0327`	`U+0043 U+0327` — `is_confusable` says true	`U+0043`
`U+00A5 U+0300`	`U+0059 U+0300`	`U+1EF2`

For a primitive whose whole job is producing a skeleton two identifiers can be compared on, returning output that the library's own detector still flags is the failure that matters. An exhaustive sweep of the BMP crossed with composing marks finds 28 base characters affected, not just the two above.

Layer 1 gains normalize_confusables_fixed_cow, the borrowing form of the fixed-point fold, and the public API calls it. The owned form now delegates to it rather than carrying a second copy of the loop, so there is one implementation to keep correct. The borrow-on-no-op guarantee (#352) is unchanged: input with nothing to fold is already a fixed point, so the common case still never allocates.

Guarded at every level: spot cases on the Layer-2 API, parity tests in the Node, Ruby, Java, Kotlin and C-ABI suites, and a Tier-3 sweep over the BMP × composing marks for both idempotence and residual confusability. The Tier-3 entry is deliberately separate from #523's lib-level sweep, because that sweep tests Layer 1 — testing the layer below the one the bindings call is how this survived a year.

⁰ and ᵒ now fold; a block-range gap had been dropping them (#590). is_latin_or_common in gen_confusables.py enumerates Latin block ranges and jumps from 0x007F to 0x00C0, leaving U+0080–U+00BF uncovered. º (U+00BA) is category Lo with Script=Latin and lives in that hole, so filter_direct read it as a non-Latin target and discarded both upstream rows pointing at it.

The symptom was an asymmetry between the only two superscript digits upstream TR39 carries — it lists visual lookalikes, and no letter resembles a superscript four:

	before	now
`⁰` U+2070	unmapped, passed through	`0` (numeric), `º` (tr39)
`⁹` U+2079	`9` (numeric), `ꝰ` (tr39)	unchanged

⁹ targets ꝰ in Latin Extended-D, so its row survived and the #89 digit rule rewrote it to the ASCII digit. ⁰ was discarded before that rule could run. With the row restored, ⁰ gains a tr39 override too, so the pair is now symmetric.

The fix admits only the three Latin letters in the gap — ª µ º — and gives ASCII_FOLD the two ordinal indicators, keeping #341's ASCII contract. Opening the whole range instead would pull in 58 rows targeting punctuation and symbols (·, °, ¶, ©), which is what that contract exists to prevent. is_latin, the source-side predicate, has the identical gap; closing it there was measured to change nothing, since no upstream row uses those three as a source, so it is documented rather than changed blind.

Table sizes move accordingly: Latin 2,181 → 2,183 mappings, tr39 overrides 45 → 46. Every documented count was re-measured against the regenerated tables rather than adjusted by hand, which turned up two that were already stale before this change — the Latin table was described as ~2,063 mappings (actual 2,183) and the Cyrillic as ~1,369 (actual 1,349).

tests/test_doc_table_counts.py existed to prevent exactly that drift, and the two files it gates were accurate. The three surfaces carrying the same figure were not gated, and had drifted by 118 rows unnoticed: src/tables/confusables_data.rs, python/disarm/_api.py, and the target-script table in the confusables user guide. All three are now gated against the same source of truth, taking the check from 5 figures to 11.

Restored the 1-argument disarm_analyze_hostname C ABI (#580). #562 widened it to take contractions, but that symbol shipped in 0.13.0 and callers are linked against the 1-argument form — widening it breaks them at link time. The contraction pass moved to a new disarm_analyze_hostname_opts(host, contractions), with the original delegating to it, matching disarm_transliterate / _opts and disarm_normalize_confusables / _opts.

Found by the header drift gate added in this same change, on its first real run against a moved main — which is the argument for the gate. Nothing else in CI could see it: the smoke test regenerates the header and compiles against it in one step, so a widened signature plus a matching call-site change is self-consistent and passes.

sanitize_filename is now a fixed point on the first pass (#570). The trailing-dot trim ran in finalize_name, after the extension split it invalidates. Input ending in a . — literal, or produced by transliteration (· U+00B7, … U+2026) — had that dot taken as the "extension", leaving an earlier dot inside the stem; trimming it then moved the boundary, so the next call split elsewhere and stripped a separator that had become stem-trailing. sanitize_filename("a*.b.") gave "a_.b", then "a.b".

The trim now runs before the split, so the boundary the split sees is the one the output has. finalize_name is unchanged and still required — the extension branch re-prepends '.', and it owns the empty / "." / ".." fallback.

The guarantee asserted is stronger than idempotence: a caller sanitizes once, so if the first pass returned something a second pass would change, the single-pass answer was already wrong. Two systems sanitizing a different number of times derived different filenames from one input, which defeats dedup on sanitized names.

Found by the Hypothesis tier, which ran nowhere automatic — see the nightly workflow below.

Restored a Kotlin test lost in a rebase. DisarmKtTest.coverageIntrospection, added with the coverage-introspection API (#563), was silently dropped when that branch was rebased through a 14-file conflict. The Kotlin source survived, so nothing failed to compile and the loss was invisible until the JVM surface was audited for this change.
16 confusable rows the generator was silently dropping (#558). scripts/gen_confusables.py's filter_latin_homoglyphs pass required the TR39 prototype to be a single basic ASCII letter. That quietly excluded every Latin-script letter whose prototype is an ASCII digit or punctuation mark — Ʒ→3, Ȣ→8, Ꝯ→9, ǃ→!, Ɂ→?, ꝸ→&, ꞉→:, ꞌ→' and eight more. Nothing distinguished them from the þ→p / ſ→f rows already in the table except the category of the target, so this was a table gap rather than a policy decision. The predicate is now is_basic_ascii_graphic and the 16 rows are folded.

This is the letter-impersonates-a-digit direction only. The reverse — a digit source folding to a look-alike letter — is still guarded by enforce_digit_target (#439); normalize_confusables("०") remains "0".

Whitespace is deliberately excluded from the widening: TR39 folds the whole Zs/Zl/Zp family to a space, but collapse_whitespace already owns that from an explicit core-defined set (#433), and a second copy in the confusables table would be a divergent duplicate of the whitespace policy.

The remaining residue is now triaged and written down in docs/provenance.md rather than inferred: 5 deliberate ASCII skeleton divergences, 16 whitespace rows owned elsewhere, and ~4,300 sources whose upstream target is non-Latin, for which a to-Latin table is the wrong home. unmapped_confusables() (#563) makes the split recomputable at any time, so the closed gap cannot silently reopen.

Documentation¶

Documented what the cleaning presets do to non-Latin text, and settled whether the confusable fold should be scoped (#624). docs/limitations.md carried the right caveat for exactly one destructive step — designed for security contexts, should not be applied to body text — and the two others never got it. No behaviour changed; what changed is that a caller can now find out before pointing one at a sentence.

Three mechanisms, orthogonal, measured on 13 scripts:

	what it does	which samples
`strip_accents`	deletes Indic vowel signs and viramas	Devanagari, Bengali, Tamil, Telugu, Kannada, Malayalam, Gujarati, Khmer
to-Latin confusable fold	splices Latin letters in	Arabic, Persian, Hebrew, Greek, Telugu, Malayalam
format-character strip	removes the ZWNJ Persian requires	Persian

A Latin acute and a Devanagari vowel sign are both category Mn, so strip_accents removes both — but in Latin an Mn is decoration and in an Indic script it carries the vowel. José → Jose is readable; বাংলা → বল is not a word. The measurable difference is length: removing an accent from precomposed Latin or Greek keeps the code point count, removing an Indic vowel sign shortens the word.

#564's escape hatch does not exist for six of these samples. For accented Latin a caller can reach past the bundle to normalize_confusables and keep the accents. For Arabic, Persian, Hebrew, Greek, Telugu and Malayalam the primitive is the destructive step: العربية → lلعربية, עברית → עבר'ת, Ελληνικά → Eλλnvikά, జ్ఞానం → జ్ఞానo. 22 Arabic code points fold to ASCII, 12 Hebrew, 65 Greek.

The fold stays unconditional, and that is now a measured decision rather than an open question. The issue asked whether canonicalize should fold non-Latin toward Latin at all. Skipping the fold when the input contains no Latin looks free — every CVE probe that needs it contains Latin — but Latin is the pivot alphabet, not the threat. Cyrillic оо and Greek οο contain no Latin, are different strings, and collide only because both fold to oo. A presence-of-Latin gate would let one impersonate the other.

Also: has_anomalies and is_mixed_script are silent on every sample, correctly — ordinary Telugu is not an attack — so a screen-then-clean pipeline gets no warning first. The clean unconditionally rule on the CVE page is now scoped to identifiers, hostnames, filenames and log lines, where every row on that page lives.

strip_obfuscation's docstring already said "non-Latin scripts that have no Latin confusable equivalent pass through unchanged", which is true and reads as reassurance while excluding exactly the case that bites. It now says what the exclusion covers. Caveats added to strip_accents, canonicalize, canonicalize_strict and strip_obfuscation on both the Rust and Python surfaces.

Held by tests/test_non_latin_fidelity.py, which derives the affected sets from behaviour rather than listing them — a hand-written list had already gone stale while the file was being written, missing that Malayalam's anusvara folds to o exactly as Telugu's does.

cargo doc is warning-free again. Six broken rustdoc links, three of them added by #620's find_key_collisions — ErrorKind is not in scope inside api, and MAX_BATCH_SIZE is pub(crate), so the published docs.rs page had dead links. cargo doc is not a CI step, which is why nothing caught them. The other three predate #620 and are the identical defect (public docs pointing at crate::hostname:: / crate::whitespace:: private paths instead of the crate::api:: re-exports); repointed while in the area.
The required status checks are named correctly again. CONTRIBUTING.md, AGENTS.md and SECURITY.md told contributors to wait for "Rust checks passed" and "Python checks passed". #583 collapsed those contexts into a single roll-up, so neither exists: branch protection requires "All checks passed", "DCO sign-off" and "iai estimated-cycles gate". A contributor following the old text waits for a check that will never report.
The four drift gates are described in one place. CONTRIBUTING.md → Drift gates now names what each guards and when it fails: the committed disarm.h (#580), the 11 documented row counts in test_doc_table_counts.py (#591), the build.rs ASCII assertions (extended to the tr39 overrides earlier in this release by #587), and JvmSignatureTest over the published Kotlin JVM signatures (#588). Each entry names the CI check that reports it, so a red run leads straight to the gate. Documentation only — no gate changes behaviour here. Two of them read a build product rather than source text, which is why they catch what source-level assertions miss.

It also records the habit those gates exist to enforce: when you regenerate a table, read the data diff, not just the test output. A generator change can silently remove rows and leave the suite green — which is how an over-broad filter deleted Ç → C during #593.

The Tier 3 listing matches tier3.yml again. CONTRIBUTING.md documented two of the five steps the workflow runs; exhaustive_grapheme (#174), exhaustive_confusables (#586) and the lib-level ignored sweep were missing. AGENTS.md gained the confusables entry and the note on why it is deliberately separate from the Layer-1 sweep.
Accented-Latin fidelity is a strip_obfuscation property, not a confusables property (#564). normalize_confusables preserves accented Latin where strip_obfuscation destroys it, at identical homoglyph recovery — because strip_accents sits in the strip_obfuscation bundle, not in the confusable primitive. Nothing in the docs said so, so a reader could reasonably conclude that disarm destroys accented Latin as a matter of course, and a benchmark cell measuring the bundle could be read as measuring the fold.

docs/security/adversarial-defense.md gains a "What each entry point costs you" section: the worked comparison, the structural explanation, and a threat-model → entry-point → cost table covering all six entry points. The confusables user guide gains the short version with a cross-link. Both are doctested, and tests/test_accented_latin_fidelity.py pins the claims — including that the loss is attributable to strip_accents specifically, and that ml_normalize folds no confusables at any fold_case setting, so it is not a homoglyph defence.

Filed and fixed together with #559 because both have the same shape: a destructive step baked into a bundle with no documented route to the non-destructive path.