Predicates¶

Return the set of Unicode scripts present in text, in order of first appearance.

Examples:

>>> detect_scripts("Hello")
[Script.LATIN]
>>> detect_scripts("Hello Мир")
[Script.LATIN, Script.CYRILLIC]

Inspect how lang="auto" would resolve for the given text.

Use this to audit or log the detection decision made by the three-stage auto-detection pipeline.

Examples:

>>> inspect_auto_lang("Київ")["chosen_lang"]
'uk'
>>> inspect_auto_lang("Москва")["reason"]
'script_default'

True if text contains characters from more than one Unicode writing system.

Resolves the UTS #39 §5.1 augmented script sets (#776), so a script pair that one writing system uses is not "mixed":

================================== ============================ Han + Hiragana + Katakana Japanese Han + Hangul Korean Han + Bopomofo Chinese ================================== ============================

So 日本語テスト is one writing system, not three scripts. Anything without a writing system in common is still mixed, including a CJK script beside a non-CJK one — 例えa is Japanese and Latin, and that is the case this check exists for.

Examples:

>>> is_mixed_script("Hello")
False
>>> is_mixed_script("Hello Мир")  # Latin + Cyrillic
True
>>> is_mixed_script("日本語テスト")  # Han + Katakana, one writing system
False
>>> is_mixed_script("ひら한")  # Japanese + Korean, no set in common
True

This is a string-level question, and callers compose it into a different one (#901). Bilingual text triggers it by design. A reject rule written as is_mixed_script(x) or has_bidi_conflict(x) or find_confusables(x) turns away every bilingual user — measured, it rejects all four bilingual strings in #901's table alongside the two spoofs. has_anomalies is the composition that tells them apart; per_word=True is that distinction on its own.

per_word splits on words, not whitespace tokens: IT-специалист, email:почта and user@почта.рф are ordinary text, and a whitespace split reports all three. It uses the detector's own splitter, so the two agree.

True if text mixes strong left-to-right and strong right-to-left characters.

This is the precondition for Unicode Bidi display-reordering (UAX #9) — the structural signal behind "BiDi Swap"-style spoofs, where an LTR brand label sits beside an RTL domain (e.g. "varonis.com.ו.קום"). Unlike a bidi-override (U+202x) check, it fires on the real letters: Latin / Cyrillic / Greek / CJK are left-to-right; Hebrew / Arabic / Syriac / Thaana / N'Ko are right-to-left; digits, punctuation and combining marks are neutral and never create a conflict on their own.

A False result is not a safety guarantee.

has_bidi_conflict("hello שלום")            True
inspect_anomalies("hello שלום").kinds      []
has_bidi_conflict("helloשלום")             True
inspect_anomalies("helloשלום").kinds       ['bidi_mixed']

Examples:

>>> has_bidi_conflict("hello")
False
>>> has_bidi_conflict("helloא")  # Latin + Hebrew
True
>>> has_bidi_conflict("hello שלום")  # whole string, so the space is no barrier
True
>>> inspect_anomalies("hello שלום").kinds  # per token, so it is two clean words
[]
>>> has_bidi_conflict("invoice\u202Egpj.exe")  # RLO override, not letters
False
>>> inspect_anomalies("invoice\u202Egpj.exe").kinds  # this is the check
['bidi']

This is a string-level question, and callers compose it into a different one (#901). Bilingual text triggers it by design. A reject rule written as is_mixed_script(x) or has_bidi_conflict(x) or find_confusables(x) turns away every bilingual user — measured, it rejects all four bilingual strings in #901's table alongside the two spoofs. has_anomalies is the composition that tells them apart; per_word=True is that distinction on its own.

per_word splits on words, not whitespace tokens: IT-специалист, email:почта and user@почта.рф are ordinary text, and a whitespace split reports all three. It uses the detector's own splitter, so the two agree.

True if text contains characters confusable with target-script characters.

Printable ASCII is never a detection (#957). ", the backtick and | are TR39 confusable sources and the fold rewrites all three — deliberately, and recorded under Five surfaces rewrite printable ASCII in the limitations page. Counting them here made this return True for every quoted sentence and every JSON document; 588 of the 1,342 pure-ASCII lines of this repository's own prose fired. The rows stay in the fold and stop being reported, so normalize_confusables('|') is still 'l' while is_confusable('|') is False.

Examples:

>>> is_confusable("pаypal")  # Cyrillic а looks like Latin a
True
>>> is_confusable("paypal")  # all genuine Latin
False

Every upstream confusable source disarm's bundled table does not fold (#563).

Read this as exposure, not as a score. A tool at 95% per-source coverage is not 95% safe — it is one query away from the other 5%, and this set is where an adaptive attacker goes when the mapped sources stop working.

Most of the set is out of scope rather than missing: a source whose upstream target is non-Latin has no business in the to-Latin table. Cross-reference CONFUSABLES_VERSION and docs/provenance.md before reading any one codepoint as a defect.

The population is TR39's, not the world's (#738). This enumerates the 6,565 single-code-point sources in confusables_upstream_sources.tsv — what upstream lists and disarm drops. A pair upstream never listed is outside the denominator as well as outside the table: U+4E28 and U+3021 score higher against l than most of TR39 in a measured font survey, and neither appears here. THREAT_MODEL.md's normalization is enumerate-the-known is the honest reading; this is not a complete gap report. It is also single-code-point by construction, so the multi-character direction (rn → m, vv → w, cl → d) has no denominator at all — disarm's answer there is three contraction rows, reachable only from is_suspicious_hostname with contractions=True, against a measured population of 571,753 bigram-to-character pairs, most of them not registrable.

The set includes five ASCII characters — %, 0, 1, I and m. TR39 is a skeleton transform (m→rn, I/1→l, 0→O), so those are upstream sources; disarm does not apply those rows because folding a legitimate ASCII m to rn corrupts prose. Nothing is filtered out here: a coverage report that quietly drops rows reads as coverage it does not have.

Examples:

>>> unmapped = unmapped_confusables()
>>> "а" in unmapped  # Cyrillic а IS folded, so it is not exposure
False
>>> "m" in unmapped  # TR39 skeleton source m→rn, deliberately not applied
True

Find confusable sources in text that disarm's table does not fold (#563).

The confusables analogue of find_untranslatable, and it follows the same convention: (character, byte_offset) pairs in order of appearance. This is what turns unmapped_confusables from a global number into something answerable against your own traffic.

Composition runs exactly as it does in normalize_confusables, so a decomposed homoglyph whose precomposed form is mapped counts as covered rather than as a gap — otherwise the report would disagree with what the transform does. Offsets are anchored in text, never in the composed intermediate.

Ordinary English will report the letter m; see unmapped_confusables for why that is deliberate.

Examples:

>>> find_unmapped_confusables("pаypal")  # Cyrillic а folds — covered
[]
>>> find_unmapped_confusables("hello")
[]

TR39 sources whose prototype is in script, and how many disarm folds (#963).

The denominator unmapped_confusables does not have. That function measures one bundled table against the whole 6,565-source population, which is the right question for a target disarm ships and a misleading one for a script it does not: Greek reports almost the entire population unmapped, and the number means only "there is no Greek table". A count determined by a table's absence is a blind spot with a number in front of it, which is worse than the exception it replaced, because it looks like data.

This is the fair figure — of the sources whose prototype is in this script, how many does disarm reach:

Examples:

>>> confusable_coverage("Greek")["sources"]
159
>>> confusable_coverage("Han")["folded"]  # 1,393 sources, no CJK fold table
0
>>> confusable_coverage(Script.THAANA)  # a script TR39 never targets
{'script': 'Thaana', 'sources': 0, 'folded': 0}

True if all characters are in U+0000–U+007F.

Examples:

>>> is_ascii("hello 123")
True
>>> is_ascii("café")
False

Which of values reduce to the same identity key (#620).

Every other disarm detector is a single-string predicate, and a collision is not a property of a single string — groß.txt is an ordinary German filename, and аdmin is only a problem next to admin. This is the set-shaped question: given these names, which of them are the same name?

That is what node-tar's PathReservations guard failed to ask before extracting two paths in parallel (CVE-2026-23950), and what a registry has to ask before accepting a second admin (CVE-2013-7236). The two want opposite policies from the same answer — one refuses the batch, the other refuses the registration — so this reports and decides nothing.

Choosing key is choosing the policy, and there is no default. Measured against the four collision CVEs in the validation matrix:

============================ ========== ========== ========= ========= key 2026-23950 2019-19844 2013-7236 2020-12063 ============================ ========== ========== ========= ========= "fold_case" yes -- -- -- "search_key" yes yes yes yes "catalog_key" yes yes yes yes "canonicalize" -- yes yes yes "canonicalize_strict" -- yes yes yes "normalize_confusables" -- yes yes yes ============================ ========== ========== ========= =========

A stronger key finds more collisions, including ones nobody attacked: search_key collides Muller with Müller and Ivan with Иван. That is not a false positive — they really are one key — it is the cost of the key you chose. sort_key is deliberately not offered: a sort key exists to collide, so reporting its collisions would be noise.

Reducing and grouping happen in one pass over one reducer, so the report cannot disagree with the collapse it describes. A group is returned only when it holds two or more distinct inputs — the same string twice is the same name twice, which a reservation table already handles.

The return is not a partition, and the two counts do not add (#763). A name that collides with nothing never appears, so the groups do not cover the input. The quantity a registry actually wants — after reduction, how many distinct identities does this batch hold? — has to be derived, and there is one correct spelling::

reduced = len(set(values)) - sum(len(g.values) for g in groups) + len(groups)

values and indices have different denominators by design (see KeyCollision), so they must never be arithmetically combined. Substituting g.indices for g.values above, or len(values) for len(set(values)), gives a formula that is right on every duplicate-free batch and wrong the moment an input repeats. Measured over 400 duplicate-free batches all four spellings agree with the truth; over 400 with one repeat injected, only this one does.

Examples:

>>> found = find_key_collisions(
...     ["groß.txt", "gross.txt", "other.txt"], key="fold_case"
... )
>>> found[0].key
'gross.txt'
>>> found[0].values
['groß.txt', 'gross.txt']
>>> found[0].indices
[0, 1]
>>> find_key_collisions(["a.txt", "b.txt"], key="fold_case")
[]

A repeated input — the shape every other example omits, and the only shape that separates the correct derivation from its three near-misses:

>>> names = ["admin", "admin", "Admin"]
>>> groups = find_key_collisions(names, key="fold_case")
>>> groups[0].values          # distinct inputs: two
['admin', 'Admin']
>>> groups[0].indices         # occurrences: three
[0, 1, 2]
>>> len(set(names)) - sum(len(g.values) for g in groups) + len(groups)
1

Three names, one identity. The three near-misses give 2, 0 and 1 — the last by cancellation rather than by construction.

True if text is a stable identity key under case folding.

Answers fold_case(text) == text.lower(). A False result says some other string folds to the same value, so a table keyed on this one can collide — groß.txt and gross.txt are the pair node-tar collided on (CVE-2026-23950), and ſtraße/straße and file/file are the same shape. Roughly 2,000 code points behave this way, including every Latin ligature, ẛ, the micro sign, and all of Cherokee (whose fold direction runs small→capital, so both cases move).

This is a fact about the string, not an accusation. groß is an ordinary German word, so a False here is not a report of an attack and the predicate is deliberately kept out of has_anomalies. What to do about it is the caller's decision: reserve both forms, reject the name, or key the table on fold_case rather than str.lower().

str.lower() is the correct comparison basis and str.casefold() is not: casefolding performs the very transform under test, so a predicate written against it is True everywhere.

Answers about disarm's own folding table (Unicode 16.0), so it also reports False for characters your Python's str.lower() knows about and that table does not — which is a collision hazard for the same reason.

A True result is not a uniqueness guarantee: two distinct stable strings can still collide under some other normalization.

Examples:

>>> is_case_fold_stable("gross.txt")
True
>>> is_case_fold_stable("groß.txt")
False
>>> is_case_fold_stable("ΟΔΟΣ")  # Greek final sigma: οδος vs οδοσ
False

Decode what a smuggled run spells, rather than reporting one is present (#701).

disarm strips the three ASCII-smuggling carriers and inspect_anomalies reports that invisible characters are there. Neither answer tells you the run reads tracked-by:acct-99213.

Presence and decode are different strengths of evidence. An invisible character can arrive by accident — a copy-paste artefact, a BOM, an editor quirk. A run that decodes to readable text cannot: random damage does not spell words. A successful decode needs no threshold and no policy to interpret, which is why it is worth reporting separately from the invisible kind.

Three schemes, all arithmetic on code point values with no table behind them:

text is populated only when the bytes are valid UTF-8 and wholly printable — meaning a reader would see it, not merely "no control character". A payload of U+202E + U+200B is valid UTF-8 with no control in it and renders as nothing, so it comes back as bytes with text=None, as does a run of arbitrary selectors. Reporting garbage would undo the reason a decode is trustworthy.

units counts the characters the run consumed, which is not the same as the carriers that carried a byte: the zero-width scheme counts its ZWJ/WJ/BOM separators and the tag scheme counts a trailing CANCEL TAG.

A well-formed emoji subdivision flag is not a payload: U+1F3F4 + tag letters + U+E007F spelling one of the three RGI values is the Scotland flag, and the allowlist used here is the stripper's own rather than a second copy of it.

percent_escape is the one scheme that is not fed to the anomaly detector. A %XX run spelling readable text is ordinary in any URL, where the three invisible carriers are never ordinary; inspect_anomalies would fire smuggled on every escaped query string. This function reports it — as evidence, decoded once — and the detector does not. %25%32%45 spells %2E, and that text is the sign of double-encoding, not a prompt to decode again.

Examples:

>>> hidden = "".join(chr(ord(c) + 0xE0000) for c in "hi")
>>> found = decode_smuggled(f"hello{hidden}")
>>> found[0].text, found[0].scheme, found[0].start
('hi', 'tag_ascii', 5)
>>> decode_smuggled("hello world")
[]

True if text is already its own canonical form under preset.

Every other normalization surface in disarm is generation path: text in, normalized text out. This is the verification path counterpart — the question to ask about bytes that arrive already bound to a decision, where quietly re-normalizing defends the comparison you are about to make and leaves the second representation free to keep circulating.

has_anomalies is not this predicate, and the gap is not small. Over every assigned code point, 142,760 of them (5,292 excluding the Private Use Area) are reported clean by the detector and are not their own canonical form — CJK compatibility ideographs, Arabic presentation forms, Kangxi radicals, fullwidth and halfwidth forms. None go the other way. An accept gate written as "reject if has_anomalies, otherwise take the bytes as given" admits every one of them.

That is not a detector bug to be fixed by widening it: ＮＨＫ is how a Japanese broadcaster writes its own name, and a detector that flags it is one callers turn off (#633, #907). Two questions, two answers — ask this one when you need canonicity, and has_anomalies when you need suspicion.

Equivalent to globals()[preset](text) == text and defined by it, but it does not build the normalized copy to answer a boolean, and nothing crosses the extension boundary but the result.

Examples:

>>> is_canonical("paypal.com")
True
>>> is_canonical("ＡＢＣ")  # fullwidth — clean to the detector
False
>>> has_anomalies("ＡＢＣ")  # ...and this is the point
False
>>> is_canonical("abc", preset="search_key")
True

True if text is already in the specified normalization form.

Examples:

>>> is_normalized("café")  # NFC by default
True
>>> is_normalized("e\u0301", form="NFC")  # NFD decomposed
False

Detect whether text contains zalgo-style combining mark abuse.

Returns True if any base character has more than threshold consecutive combining marks in NFD decomposition.

Examples:

>>> is_zalgo("café")
False
>>> is_zalgo("Việt Nam")
False
>>> is_zalgo("ḧ̸̡̢̧̛̗̱̜̼̯̞̙́̑̾̊̿̏̒̓̕ě̵̢̧̛̗̱̜̼̯̞̙̈́̑̾̊̿̏̒̓̕l̸̡̢̧̛̗̱̜̼̯̞̙̈́̑̾̊̿̏̒̓̕l̸̡̢̧̛̗̱̜̼̯̞̙̈́̑̾̊̿̏̒̓̕ơ̵̢̧̗̱̜̼̯̞̙̈́̑̾̊̿̏̒̓̕")
True

Flag a hostname as suspicious for Unicode homoglyph spoofing.

Returns (suspicious, analysis) where analysis is a HostnameAnalysis with attributes:

• suspicious: bool — True if a problem was detected (mixed-script, a bundled-table confusable, or a bidi-direction conflict). Because the confusable check is an any-character screen, this flags essentially every hostname with a non-Latin letter — legitimate (москва.рф) as well as spoofs — so it is a maximally conservative screen, not a precise verdict.
• scripts: list[str] — Unicode scripts found across all labels.
• mixed_script: bool — True if any single label contains more than one script.
• has_confusables: bool — True if confusable homoglyphs found. Read after the UTS #46 mapping and NFKC, so it cannot see a compatibility form by construction: ｇoogle.com is already google.com by the time this is computed, and False is the correct answer — after mapping there is no confusable left. Seeing canonical differ from the input while this stays False means compat_fold, not a defect.
• bidi_conflict: bool — True if the decoded hostname mixes strong left-to-right and strong right-to-left characters (the "BiDi Swap" reorder precondition). Folded into suspicious.
• bidi_control: bool — True if the decoded hostname carries a UAX #9 bidi control character: an override (U+202D/U+202E), embedding (U+202A–U+202C), isolate (U+2066–U+2069) or directional mark (U+200E/U+200F/U+061C). Disjoint from bidi_conflict, which reads strong-direction letters only and is therefore blind to the RLO extension spoof. IDNA2008 disallows every character in the set, so this is folded into suspicious and the characters are stripped from canonical.
• has_invisible: bool — True if the decoded hostname carries an invisible character of any class: zero-width (U+200B-U+200D, U+2060-U+2064, U+FEFF, U+180E), tag (U+E0000-U+E007F), variation selector (U+FE00-U+FE0F, U+E0100-U+E01EF), noncharacter (U+FDD0-U+FDEF and the last two of every plane), or private use (U+E000-U+F8FF, planes 15 and 16). Disjoint from bidi_control — these carry no direction at all, so neither bidi field can see them. RFC 5892 puts the tag, variation-selector, noncharacter and private-use classes in DISALLOWED outright, which is what justifies including private use and variation selectors here. U+200C/U+200D are the exception — CONTEXTJ, so conditionally permitted; the screen flags them anyway as a deliberate fail-closed policy. Folded into suspicious. They are removed per label before any other field is computed, so a hostname whose only non-ASCII is an invisible no longer reports a phantom script (U+FEFF sits in the Arabic Presentation Forms block, U+FDD0 in its range).
• compat_fold: bool — True if any label carried a Unicode compatibility form before normalization: fullwidth (ｇoogle), ligature (file), Roman numeral (ⅠBM), mathematical alphanumeric (𝗀𝗈𝗈𝗀𝗅𝖾), circled, superscript, and the rest of the compatibility repertoire. The predicate is RFC 5892 §2.1's, applied per code point: a character c where toNFKC(c) != c is DISALLOWED in an IDN label, so IDNA2008 disallows the whole set and this is folded into suspicious on the same footing as bidi_control and has_invisible. The threat is a blocklist bypass rather than a lookalike: ｅvil.com is absent from a blocked set, screens clean, and resolves to evil.com. Tested per character rather than "NFKC changed the label", which would fire on decomposed input that is entirely valid (한국.kr written with conjoining jamo). Read per label, not over the whole hostname: three of the four UTS #46 label separators carry a compatibility decomposition (U+FF0E and U+FF61 do, U+3002 does not), and a separator is structure rather than label content. This is the one field read from the raw input — every other field is computed after normalization, which is what makes them work and also what erases this evidence.
• cross_label_script: bool — True if the labels span more than one distinct script. Broader and noisier than bidi_conflict (it fires on benign IDN ccTLDs like google.рф), so it is not folded into suspicious; exposed for caller policy.
• label_scripts: list[list[str]] — per-label resolved scripts, left to right.
• whole_script_confusable: bool — True if any label is a whole-script confusable: single-script, non-Latin, whose confusable skeleton is entirely Latin (e.g. Cyrillic аррӏе → apple). A graded signal, not a verdict — on its own it fires on short non-Latin ccTLDs (ру→py) and on real words (оса→oca), so it is not folded into suspicious.
• label_whole_script_confusable: list[bool] — per-label flags, parallel to label_scripts, so a caller can exclude the TLD label. The precise, low-false-positive policy is wsc(non-TLD label) and TLD-is-Latin (plus a caller-supplied protected-name list for the irreducible оса-style case).
• canonical: str — Latin-normalized form of the hostname.

A hostname is flagged suspicious if any single label is mixed-script (draws on more than one Unicode script, excluding Common/Inherited), contains confusable homoglyphs, or has a bidi-direction conflict (bidi_conflict), carries a bidi control character (bidi_control), or carries a zero-width/invisible character (has_invisible), or carries a compatibility form (compat_fold). The mixed-script rule is conservative and fails closed: it flags benign combinations such as Latin+CJK as well as spoofing ones, so a caller wanting a more permissive policy can inspect the mixed_script and scripts fields and decide for itself.

A False (not-suspicious) result is not a safety guarantee. It means only that no mixed-script label and no confusable from the bundled TR39 table was found. Confusables outside the bundled table are not detected and report not-suspicious. Base allow/deny decisions on the granular findings (including whole_script_confusable) plus your own policy — a detector can attest the presence of a problem, never the absence of all problems.

Examples:

>>> suspicious, analysis = is_suspicious_hostname("google.com")
>>> suspicious
False
>>> analysis.canonical
'google.com'
>>> _s, a = is_suspicious_hostname("arnazon.com", contractions=True)
>>> a.canonical
'amazon.com'